0. HBase Filtering

使用Get或Scan操作从HBase读取数据时, 可以使用自定义过滤器将结果的子集返回给客户端. 虽然这不会减少服务器端IO, 但确实会减少网络带宽并减少客户端需要处理的数据量. 过滤器通常使用Java API使用, 但可以从HBase Shell中使用以进行测试和调试. When reading data from HBase using Get or Scan operations, you can use custom filters to return a subset of results to the client. While this does not reduce server-side IO, it does reduce network bandwidth and reduces the amount of data the client needs to process. Filters are generally used using the Java API, but can be used from HBase Shell for testing and debugging purposes.

有关HBase中获取和扫描的更多信息, 请参见 Reading Data from HBase. For more information on Gets and Scans in HBase, see Reading Data from HBase.

1. Dynamically Loading a Custom Filter 动态加载自定义过滤器

默认情况下, CDH可以通过向您的过滤器指定的目录 (hbase.dynamic.jars.dir, 默认为 HBase根目录下的 lib/ 目录)添加JAR来动态加载自定义过滤器. CDH, by default, has the ability to dynamically load a custom filter by adding a JAR with your filter to the directory specified by the hbase.dynamic.jars.dir property (which defaults to the lib/ directory under the HBase root directory).

要禁用自动加载动态JAR, 请在 hbase-site.xml 中设置 hbase.use.dynamic.jars 为 false . To disable automatic loading of dynamic JARs, set hbase.use.dynamic.jars to false in the advanced configuration snippet for hbase-site.xml if you use Cloudera Manager, or to hbase-site.xml otherwise.

2. Filter Syntax Guidelines 过滤器语法准则

HBase过滤器在括号中使用零个或多个参数. 如果参数是字符串, 则用单引号, 如 ('string') 括起来. HBase filters take zero or more arguments, in parentheses. Where the argument is a string, it is surrounded by single quotes (‘string’).

3. Logical Operators, Comparison Operators and Comparators 逻辑运算符, 比较运算符和比较器

过滤器可以与逻辑运算符组合在一起使用. 一些过滤器结合了比较运算符和比较器. Filters can be combined together with logical operators. Some filters take a combination of comparison operators and comparators. Following is the list of each.

3.1 Logical Operators 逻辑运算符

• AND - the key-value must pass both the filters to be included in the results.
• OR - the key-value must pass at least one of the filters to be included in the results.
• SKIP - for a particular row, if any of the key-values do not pass the filter condition, the entire row is skipped.
• WHILE - For a particular row, it continues to emit key-values until a key-value is reached that fails the filter condition.
• Compound Filters - Using these operators, a hierarchy of filters can be created. For example:

(Filter1 AND Filter2)OR(Filter3 AND Filter4)

3.2 Comparison Operators 比较运算符

• LESS (<)
• LESS_OR_EQUAL (<=)
• EQUAL (=)
• NOT_EQUAL (!=)
• GREATER_OR_EQUAL (>=)
• GREATER (>)
• NO_OP (no operation)

3.3 Comparators 比较器

• BinaryComparator - lexicographically compares against the specified byte array using the Bytes.compareTo(byte[], byte[]) method.
• BinaryPrefixComparator - lexicographically compares against a specified byte array. It only compares up to the length of this byte array.
• RegexStringComparator - compares against the specified byte array using the given regular expression. Only EQUAL and NOT_EQUAL comparisons are valid with this comparator.
• SubStringComparator - tests whether or not the given substring appears in a specified byte array. The comparison is case insensitive. Only EQUAL and NOT_EQUAL comparisons are valid with this comparator.

Examples

Example1: >, 'binary:abc' will match everything that is lexicographically greater than "abc"
Example2: =, 'binaryprefix:abc' will match everything whose first 3 characters are lexicographically equal to "abc"
Example3: !=, 'regexstring:ab*yz' will match everything that doesn't begin with "ab" and ends with "yz"
Example4: =, 'substring:abc123' will match everything that begins with the substring "abc123"

3.4 Compound Operators 复合运算符

Within an expression, parentheses can be used to group clauses together, and parentheses have the highest order of precedence.

• SKIP and WHILE operators are next, and have the same precedence.
• The AND operator is next.
• The OR operator is next.

Examples

A filter string of the form: "Filter1 AND Filter2 OR Filter3” will be evaluated as: "(Filter1 AND Filter2) OR Filter3”
A filter string of the form: "Filter1 AND SKIP Filter2 OR Filter3” will be evaluated as: "(Filter1 AND (SKIP Filter2)) OR Filter3”

3.5 Filter Types 过滤器类型

HBase包括几种过滤器类型, 以及将过滤器组合在一起并创建自己的自定义过滤器的功能。 HBase includes several filter types, as well as the ability to group filters together and create your own custom filters.

• KeyOnlyFilter - takes no arguments. Returns the key portion of each key-value pair.

Syntax: KeyOnlyFilter ()

• FirstKeyOnlyFilter - takes no arguments. Returns the key portion of the first key-value pair.

Syntax: FirstKeyOnlyFilter ()

• PrefixFilter - takes a single argument, a prefix of a row key. It returns only those key-values present in a row that start with the specified row prefix

Syntax:  PrefixFilter ('<row_prefix>')

Example: PrefixFilter ('Row')

• ColumnPrefixFilter - takes a single argument, a column prefix. It returns only those key-values present in a column that starts with the specified column prefix.

Syntax:  ColumnPrefixFilter ('<column_prefix>')
Example: ColumnPrefixFilter ('Col')

• MultipleColumnPrefixFilter - takes a list of column prefixes. It returns key-values that are present in a column that starts with any of the specified column prefixes.

Syntax:  MultipleColumnPrefixFilter ('<column_prefix>', '<column_prefix>', …, '<column_prefix>')
Example: MultipleColumnPrefixFilter ('Col1', 'Col2')

• ColumnCountGetFilter - takes one argument, a limit. It returns the first limit number of columns in the table.

Syntax:  ColumnCountGetFilter ('<limit>')
Example: ColumnCountGetFilter (4)

• PageFilter - takes one argument, a page size. It returns page size number of rows from the table.

Syntax:  PageFilter ('<page_size>')
Example: PageFilter (2)

• ColumnPaginationFilter - takes two arguments, a limit and offset. It returns limit number of columns after offset number of columns. It does this for all the rows.

Syntax:  ColumnPaginationFilter ('<limit>', '<offset>')
Example: ColumnPaginationFilter (3, 5)

• InclusiveStopFilter - takes one argument, a row key on which to stop scanning. It returns all key-values present in rows up to and including the specified row.

Syntax:  InclusiveStopFilter ('<stop_row_key>')
Example: InclusiveStopFilter ('Row2')

• TimeStampsFilter - takes a list of timestamps. It returns those key-values whose timestamps matches any of the specified timestamps.

Syntax:  TimeStampsFilter (<timestamp>, <timestamp>, ... ,<timestamp>)
Example: TimeStampsFilter (5985489, 48895495, 58489845945)

• RowFilter - takes a compare operator and a comparator. It compares each row key with the comparator using the compare operator and if the comparison returns true, it returns all the key-values in that row.

Syntax:  RowFilter (<compareOp>, '<row_comparator>')
Example: RowFilter (<=, 'binary:xyz)

• FamilyFilter - takes a compare operator and a comparator. It compares each family name with the comparator using the compare operator and if the comparison returns true, it returns all the key-values in that family.

Syntax:  FamilyFilter (<compareOp>, '<family_comparator>')
Example: FamilyFilter (>=, 'binaryprefix:FamilyB')

• QualifierFilter - takes a compare operator and a comparator. It compares each qualifier name with the comparator using the compare operator and if the comparison returns true, it returns all the key-values in that column.

Syntax:  QualifierFilter (<compareOp>, '<qualifier_comparator>')
Example: QualifierFilter (=, 'substring:Column1')

• ValueFilter - takes a compare operator and a comparator. It compares each value with the comparator using the compare operator and if the comparison returns true, it returns that key-value.

Syntax:  ValueFilter (<compareOp>, '<value_comparator>')
Example: ValueFilter (!=, 'binary:Value')

• DependentColumnFilter - takes two arguments required arguments, a family and a qualifier. It tries to locate this column in each row and returns all key-values in that row that have the same timestamp. If the row does not contain the specified column, none of the key-values in that row will be returned. The filter can also take an optional boolean argument, dropDependentColumn. If set to true, the column used for the filter does not get returned.

The filter can also take two more additional optional arguments, a compare operator and a value comparator, which are further checks in addition to the family and qualifier. If the dependent column is found, its value should also pass the value check. If it does pass the value check, only then is its timestamp taken into consideration.

Syntax:  DependentColumnFilter ('<family>', '<qualifier>', <boolean>, <compare operator>, '<value comparator')
         DependentColumnFilter ('<family>', '<qualifier>', <boolean>)
         DependentColumnFilter ('<family>', '<qualifier>')

Example: DependentColumnFilter ('conf', 'blacklist', false, >=, 'zebra')
         DependentColumnFilter ('conf', 'blacklist', true)
         DependentColumnFilter ('conf', 'blacklist')

• SingleColumnValueFilter - takes a column family, a qualifier, a compare operator and a comparator. If the specified column is not found, all the columns of that row will be emitted. If the column is found and the comparison with the comparator returns true, all the columns of the row will be emitted. If the condition fails, the row will not be emitted. This filter also takes two additional optional boolean arguments, filterIfColumnMissing and setLatestVersionOnly.

If the filterIfColumnMissing flag is set to true, the columns of the row will not be emitted if the specified column to check is not found in the row. The default value is false.

If the setLatestVersionOnly flag is set to false, it will test previous versions (timestamps) in addition to the most recent. The default value is true.

These flags are optional and dependent on each other. You must set neither or both of them together.

Syntax:  SingleColumnValueFilter ('<family>', '<qualifier>', <compare operator>, '<comparator>', <filterIfColumnMissing_boolean>, <latest_version_boolean>)
Syntax:  SingleColumnValueFilter ('<family>', '<qualifier>', <compare operator>, '<comparator>')

Example: SingleColumnValueFilter ('FamilyA', 'Column1', <=, 'abc', true, false)
Example: SingleColumnValueFilter ('FamilyA', 'Column1', <=, 'abc')

• SingleColumnValueExcludeFilter - takes the same arguments and behaves same as SingleColumnValueFilter. However, if the column is found and the condition passes, all the columns of the row will be emitted except for the tested column value.

Syntax:  SingleColumnValueExcludeFilter (<family>, <qualifier>, <compare operators>, <comparator>, <latest_version_boolean>, <filterIfColumnMissing_boolean>)
Syntax:  SingleColumnValueExcludeFilter (<family>, <qualifier>, <compare operator> <comparator>)

Example: SingleColumnValueExcludeFilter ('FamilyA', 'Column1', '<=', 'abc', 'false', 'true')
Example: SingleColumnValueExcludeFilter ('FamilyA', 'Column1', '<=', 'abc')

• ColumnRangeFilter - takes either minColumn, maxColumn, or both. Returns only those keys with columns that are between minColumn and maxColumn. It also takes two boolean variables to indicate whether to include the minColumn and maxColumn or not. If you don’t want to set the minColumn or the maxColumn, you can pass in an empty argument.

Syntax:  ColumnRangeFilter ('<minColumn >', <minColumnInclusive_bool>, '<maxColumn>', <maxColumnInclusive_bool>)

Example: ColumnRangeFilter ('abc', true, 'xyz', false)

• Custom Filter - You can create a custom filter by implementing the Filter class. The JAR must be available on all RegionServers. HBase Shell Example This example scans the ‘users’ table for rows where the contents of the cf:name column equals the string ‘abc’.

hbase> scan 'users', { FILTER => SingleColumnValueFilter.new(Bytes.toBytes('cf'),
      Bytes.toBytes('name'), CompareFilter::CompareOp.valueOf('EQUAL'),
      BinaryComparator.new(Bytes.toBytes('abc')))}

Java API Example This example, taken from the HBase unit test found in hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestSingleColumnValueFilter.java , shows how to use the Java API to implement several different filters..

/**
 *
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.hadoop.hbase.filter;

import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;

import java.util.regex.Pattern;

import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.SmallTests;
import org.apache.hadoop.hbase.filter.CompareFilter.CompareOp;
import org.apache.hadoop.hbase.util.Bytes;
import org.junit.Before;
import org.junit.Test;
import org.junit.experimental.categories.Category;

/**
 * Tests the value filter
 */
@Category(SmallTests.class)
public class TestSingleColumnValueFilter {
  private static final byte[] ROW = Bytes.toBytes("test");
  private static final byte[] COLUMN_FAMILY = Bytes.toBytes("test");
  private static final byte [] COLUMN_QUALIFIER = Bytes.toBytes("foo");
  private static final byte[] VAL_1 = Bytes.toBytes("a");
  private static final byte[] VAL_2 = Bytes.toBytes("ab");
  private static final byte[] VAL_3 = Bytes.toBytes("abc");
  private static final byte[] VAL_4 = Bytes.toBytes("abcd");
  private static final byte[] FULLSTRING_1 =
    Bytes.toBytes("The quick brown fox jumps over the lazy dog.");
  private static final byte[] FULLSTRING_2 =
    Bytes.toBytes("The slow grey fox trips over the lazy dog.");
  private static final String QUICK_SUBSTR = "quick";
  private static final String QUICK_REGEX = ".+quick.+";
  private static final Pattern QUICK_PATTERN = Pattern.compile("QuIcK", Pattern.CASE_INSENSITIVE | Pattern.DOTALL);

  Filter basicFilter;
  Filter nullFilter;
  Filter substrFilter;
  Filter regexFilter;
  Filter regexPatternFilter;

  @Before
  public void setUp() throws Exception {
    basicFilter = basicFilterNew();
    nullFilter = nullFilterNew();
    substrFilter = substrFilterNew();
    regexFilter = regexFilterNew();
    regexPatternFilter = regexFilterNew(QUICK_PATTERN);
  }

  private Filter basicFilterNew() {
    return new SingleColumnValueFilter(COLUMN_FAMILY, COLUMN_QUALIFIER,
      CompareOp.GREATER_OR_EQUAL, VAL_2);
  }

  private Filter nullFilterNew() {
    return new SingleColumnValueFilter(COLUMN_FAMILY, COLUMN_QUALIFIER, CompareOp.NOT_EQUAL,
        new NullComparator());
  }

  private Filter substrFilterNew() {
    return new SingleColumnValueFilter(COLUMN_FAMILY, COLUMN_QUALIFIER,
      CompareOp.EQUAL,
      new SubstringComparator(QUICK_SUBSTR));
  }

  private Filter regexFilterNew() {
    return new SingleColumnValueFilter(COLUMN_FAMILY, COLUMN_QUALIFIER,
      CompareOp.EQUAL,
      new RegexStringComparator(QUICK_REGEX));
  }

  private Filter regexFilterNew(Pattern pattern) {
    return new SingleColumnValueFilter(COLUMN_FAMILY, COLUMN_QUALIFIER,
        CompareOp.EQUAL,
        new RegexStringComparator(pattern.pattern(), pattern.flags()));
  }

  private void basicFilterTests(SingleColumnValueFilter filter)
      throws Exception {
    KeyValue kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER, VAL_2);
    assertTrue("basicFilter1", filter.filterKeyValue(kv) == Filter.ReturnCode.INCLUDE);
    kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER, VAL_3);
    assertTrue("basicFilter2", filter.filterKeyValue(kv) == Filter.ReturnCode.INCLUDE);
    kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER, VAL_4);
    assertTrue("basicFilter3", filter.filterKeyValue(kv) == Filter.ReturnCode.INCLUDE);
    assertFalse("basicFilterNotNull", filter.filterRow());
    filter.reset();
    kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER, VAL_1);
    assertTrue("basicFilter4", filter.filterKeyValue(kv) == Filter.ReturnCode.NEXT_ROW);
    kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER, VAL_2);
    assertTrue("basicFilter4", filter.filterKeyValue(kv) == Filter.ReturnCode.NEXT_ROW);
    assertFalse("basicFilterAllRemaining", filter.filterAllRemaining());
    assertTrue("basicFilterNotNull", filter.filterRow());
    filter.reset();
    filter.setLatestVersionOnly(false);
    kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER, VAL_1);
    assertTrue("basicFilter5", filter.filterKeyValue(kv) == Filter.ReturnCode.INCLUDE);
    kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER, VAL_2);
    assertTrue("basicFilter5", filter.filterKeyValue(kv) == Filter.ReturnCode.INCLUDE);
    assertFalse("basicFilterNotNull", filter.filterRow());
  }

  private void nullFilterTests(Filter filter) throws Exception {
    ((SingleColumnValueFilter) filter).setFilterIfMissing(true);
    KeyValue kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER, FULLSTRING_1);
    assertTrue("null1", filter.filterKeyValue(kv) == Filter.ReturnCode.INCLUDE);
    assertFalse("null1FilterRow", filter.filterRow());
    filter.reset();
    kv = new KeyValue(ROW, COLUMN_FAMILY, Bytes.toBytes("qual2"), FULLSTRING_2);
    assertTrue("null2", filter.filterKeyValue(kv) == Filter.ReturnCode.INCLUDE);
    assertTrue("null2FilterRow", filter.filterRow());
  }

  private void substrFilterTests(Filter filter)
      throws Exception {
    KeyValue kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER,
      FULLSTRING_1);
    assertTrue("substrTrue",
      filter.filterKeyValue(kv) == Filter.ReturnCode.INCLUDE);
    kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER,
      FULLSTRING_2);
    assertTrue("substrFalse", filter.filterKeyValue(kv) == Filter.ReturnCode.INCLUDE);
    assertFalse("substrFilterAllRemaining", filter.filterAllRemaining());
    assertFalse("substrFilterNotNull", filter.filterRow());
  }

  private void regexFilterTests(Filter filter)
      throws Exception {
    KeyValue kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER,
      FULLSTRING_1);
    assertTrue("regexTrue",
      filter.filterKeyValue(kv) == Filter.ReturnCode.INCLUDE);
    kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER,
      FULLSTRING_2);
    assertTrue("regexFalse", filter.filterKeyValue(kv) == Filter.ReturnCode.INCLUDE);
    assertFalse("regexFilterAllRemaining", filter.filterAllRemaining());
    assertFalse("regexFilterNotNull", filter.filterRow());
  }

  private void regexPatternFilterTests(Filter filter)
      throws Exception {
    KeyValue kv = new KeyValue(ROW, COLUMN_FAMILY, COLUMN_QUALIFIER,
      FULLSTRING_1);
    assertTrue("regexTrue",
      filter.filterKeyValue(kv) == Filter.ReturnCode.INCLUDE);
    assertFalse("regexFilterAllRemaining", filter.filterAllRemaining());
    assertFalse("regexFilterNotNull", filter.filterRow());
  }

  private Filter serializationTest(Filter filter)
      throws Exception {
    // Decompose filter to bytes.
    byte[] buffer = filter.toByteArray();

    // Recompose filter.
    Filter newFilter = SingleColumnValueFilter.parseFrom(buffer);
    return newFilter;
  }

  /**
   * Tests identification of the stop row
   * @throws Exception
   */
  @Test
  public void testStop() throws Exception {
    basicFilterTests((SingleColumnValueFilter) basicFilter);
    nullFilterTests(nullFilter);
    substrFilterTests(substrFilter);
    regexFilterTests(regexFilter);
    regexPatternFilterTests(regexPatternFilter);
  }

  /**
   * Tests serialization
   * @throws Exception
   */
  @Test
  public void testSerialization() throws Exception {
    Filter newFilter = serializationTest(basicFilter);
    basicFilterTests((SingleColumnValueFilter)newFilter);
    newFilter = serializationTest(nullFilter);
    nullFilterTests(newFilter);
    newFilter = serializationTest(substrFilter);
    substrFilterTests(newFilter);
    newFilter = serializationTest(regexFilter);
    regexFilterTests(newFilter);
    newFilter = serializationTest(regexPatternFilter);
    regexPatternFilterTests(newFilter);
  }

}

原文:https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/admin_hbase_filtering.html