hadoop/src/contrib/hbase/CHANGES.txt

HBase Change Log

Release 0.16.0 - 2008-02-04

  INCOMPATIBLE CHANGES
   HADOOP-2056 A table with row keys containing colon fails to split regions
   HADOOP-2079 Fix generated HLog, HRegion names
   HADOOP-2495 Minor performance improvements: Slim-down BatchOperation, etc. 
   HADOOP-2506 Remove the algebra package
   HADOOP-2519 Performance improvements: Customized RPC serialization
   HADOOP-2478 Restructure how HBase lays out files in the file system (phase 1)
               (test input data)
   HADOOP-2478 Restructure how HBase lays out files in the file system (phase 2)
               Includes migration tool org.apache.hadoop.hbase.util.Migrate
   HADOOP-2558 org.onelab.filter.BloomFilter class uses 8X the memory it should
               be using

  NEW FEATURES
    HADOOP-2061 Add new Base64 dialects
    HADOOP-2084 Add a LocalHBaseCluster
    HADOOP-2068 RESTful interface (Bryan Duxbury via Stack)
    HADOOP-2316 Run REST servlet outside of master
                (Bryan Duxbury & Stack)
    HADOOP-1550 No means of deleting a'row' (Bryan Duxbuery via Stack)
    HADOOP-2384 Delete all members of a column family on a specific row
                (Bryan Duxbury via Stack)
    HADOOP-2395 Implement "ALTER TABLE ... CHANGE column" operation
                (Bryan Duxbury via Stack)
    HADOOP-2240 Truncate for hbase (Edward Yoon via Stack)
    HADOOP-2389 Provide multiple language bindings for HBase (Thrift)
                (David Simpson via Stack)

  OPTIMIZATIONS
   HADOOP-2479 Save on number of Text object creations
   HADOOP-2485 Make mapfile index interval configurable (Set default to 32
               instead of 128)
   HADOOP-2553 Don't make Long objects calculating hbase type hash codes
   HADOOP-2377 Holding open MapFile.Readers is expensive, so use less of them
   HADOOP-2407 Keeping MapFile.Reader open is expensive: Part 2
   HADOOP-2533 Performance: Scanning, just creating MapWritable in next
               consumes >20% CPU
   HADOOP-2443 Keep lazy cache of regions in client rather than an
               'authoritative' list (Bryan Duxbury via Stack)
   HADOOP-2600 Performance: HStore.getRowKeyAtOrBefore should use
               MapFile.Reader#getClosest (before)
               (Bryan Duxbury via Stack)

  BUG FIXES
   HADOOP-2059 In tests, exceptions in min dfs shutdown should not fail test
               (e.g. nightly #272)
   HADOOP-2064 TestSplit assertion and NPE failures (Patch build #952 and #953)
   HADOOP-2124 Use of `hostname` does not work on Cygwin in some cases
   HADOOP-2083 TestTableIndex failed in #970 and #956
   HADOOP-2109 Fixed race condition in processing server lease timeout.
   HADOOP-2137 hql.jsp : The character 0x19 is not valid
   HADOOP-2109 Fix another race condition in processing dead servers,
               Fix error online meta regions: was using region name and not
               startKey as key for map.put. Change TestRegionServerExit to
               always kill the region server for the META region. This makes
               the test more deterministic and getting META reassigned was
               problematic.
   HADOOP-2155 Method expecting HBaseConfiguration throws NPE when given Configuration
   HADOOP-2156 BufferUnderflowException for un-named HTableDescriptors
   HADOOP-2161 getRow() is orders of magnitudes slower than get(), even on rows
               with one column (Clint Morgan and Stack)
   HADOOP-2040 Hudson hangs AFTER test has finished
   HADOOP-2274 Excess synchronization introduced by HADOOP-2139 negatively
               impacts performance
   HADOOP-2196 Fix how hbase sits in hadoop 'package' product
   HADOOP-2276 Address regression caused by HADOOP-2274, fix HADOOP-2173 (When
               the master times out a region servers lease, the region server
               may not restart)
   HADOOP-2253 getRow can return HBASE::DELETEVAL cells
               (Bryan Duxbury via Stack)
   HADOOP-2295 Fix assigning a region to multiple servers
   HADOOP-2234 TableInputFormat erroneously aggregates map values
   HADOOP-2308 null regioninfo breaks meta scanner
   HADOOP-2304 Abbreviated symbol parsing error of dir path in jar command
               (Edward Yoon via Stack)
   HADOOP-2320 Committed TestGet2 is managled (breaks build).
   HADOOP-2322 getRow(row, TS) client interface not properly connected
   HADOOP-2309 ConcurrentModificationException doing get of all region start keys
   HADOOP-2321 TestScanner2 does not release resources which sometimes cause the
               test to time out
   HADOOP-2315 REST servlet doesn't treat / characters in row key correctly
               (Bryan Duxbury via Stack)
   HADOOP-2332 Meta table data selection in Hbase Shell
               (Edward Yoon via Stack)
   HADOOP-2347 REST servlet not thread safe but run in a threaded manner
               (Bryan Duxbury via Stack)
   HADOOP-2365 Result of HashFunction.hash() contains all identical values
   HADOOP-2362 Leaking hdfs file handle on region split
   HADOOP-2338 Fix NullPointerException in master server.
   HADOOP-2380 REST servlet throws NPE when any value node has an empty string
               (Bryan Duxbury via Stack)
   HADOOP-2350 Scanner api returns null row names, or skips row names if
               different column families do not have entries for some rows
   HADOOP-2283 AlreadyBeingCreatedException (Was: Stuck replay of failed
               regionserver edits)
   HADOOP-2392 TestRegionServerExit has new failure mode since HADOOP-2338
   HADOOP-2324 Fix assertion failures in TestTableMapReduce
   HADOOP-2396 NPE in HMaster.cancelLease
   HADOOP-2397 The only time that a meta scanner should try to recover a log is
               when the master is starting
   HADOOP-2417 Fix critical shutdown problem introduced by HADOOP-2338
   HADOOP-2418 Fix assertion failures in TestTableMapReduce, TestTableIndex,
               and TestTableJoinMapReduce
   HADOOP-2414 Fix ArrayIndexOutOfBoundsException in bloom filters.
   HADOOP-2430 Master will not shut down if there are no active region servers
   HADOOP-2199 Add tools for going from hregion filename to region name in logs
   HADOOP-2441 Fix build failures in TestHBaseCluster
   HADOOP-2451 End key is incorrectly assigned in many region splits
   HADOOP-2455 Error in Help-string of CREATE command (Edward Yoon via Stack)
   HADOOP-2465 When split parent regions are cleaned up, not all the columns are
               deleted
   HADOOP-2468 TestRegionServerExit failed in Hadoop-Nightly #338
   HADOOP-2467 scanner truncates resultset when > 1 column families
   HADOOP-2503 REST Insert / Select encoding issue (Bryan Duxbury via Stack)
   HADOOP-2505 formatter classes missing apache license
   HADOOP-2504 REST servlet method for deleting a scanner was not properly
               mapped (Bryan Duxbury via Stack)
   HADOOP-2507 REST servlet does not properly base64 row keys and column names
               (Bryan Duxbury via Stack)
   HADOOP-2530 Missing type in new hbase custom RPC serializer
   HADOOP-2490 Failure in nightly #346 (Added debugging of hudson failures).
   HADOOP-2558 fixes for build up on hudson (part 1, part 2, part 3, part 4)
   HADOOP-2500 Unreadable region kills region servers
   HADOOP-2579 Initializing a new HTable object against a nonexistent table
               throws a NoServerForRegionException instead of a
               TableNotFoundException when a different table has been created
               previously (Bryan Duxbury via Stack)
   HADOOP-2587 Splits blocked by compactions cause region to be offline for
               duration of compaction. 
   HADOOP-2592 Scanning, a region can let out a row that its not supposed
               to have
   HADOOP-2493 hbase will split on row when the start and end row is the
               same cause data loss (Bryan Duxbury via Stack)
   HADOOP-2629 Shell digests garbage without complaint
   HADOOP-2619 Compaction errors after a region splits
   HADOOP-2621 Memcache flush flushing every 60 secs with out considering
               the max memcache size
   HADOOP-2584 Web UI displays an IOException instead of the Tables
   HADOOP-2650 Remove Writables.clone and use WritableUtils.clone from
               hadoop instead
   HADOOP-2668 Documentation and improved logging so fact that hbase now
               requires migration comes as less of a surprise
   HADOOP-2686 Removed tables stick around in .META.
   HADOOP-2688 IllegalArgumentException processing a shutdown stops
               server going down and results in millions of lines of output
   HADOOP-2706 HBase Shell crash
   HADOOP-2712 under load, regions won't split
   HADOOP-2675 Options not passed to rest/thrift
   HADOOP-2722 Prevent unintentional thread exit in region server and master
   HADOOP-2718 Copy Constructor HBaseConfiguration(Configuration) will override
               hbase configurations if argumant is not an instance of
               HBaseConfiguration.
   HADOOP-2753 Back out 2718; programmatic config works but hbase*xml conf
               is overridden
   HADOOP-2718 Copy Constructor HBaseConfiguration(Configuration) will override
               hbase configurations if argumant is not an instance of
               HBaseConfiguration (Put it back again).
   HADOOP-2631 2443 breaks HTable.getStartKeys when there is more than one
               table or table you are enumerating isn't the first table
   Delete empty file: src/contrib/hbase/src/java/org/apache/hadoop/hbase/mapred/
               TableOutputCollector.java per Nigel Daley
   HADOOP-2731 Under load, regions become extremely large and eventually cause
               region servers to become unresponsive
   HADOOP-2773 Master marks region offline when it is recovering from a region
               server death

  IMPROVEMENTS
   HADOOP-2401 Add convenience put method that takes writable
               (Johan Oskarsson via Stack)
   HADOOP-2074 Simple switch to enable DEBUG level-logging in hbase
   HADOOP-2088 Make hbase runnable in $HADOOP_HOME/build(/contrib/hbase)
   HADOOP-2126 Use Bob Jenkins' hash for bloom filters
   HADOOP-2157 Make Scanners implement Iterable
   HADOOP-2176 Htable.deleteAll documentation is ambiguous
   HADOOP-2139 (phase 1) Increase parallelism in region servers.
   HADOOP-2267 [Hbase Shell] Change the prompt's title from 'hbase' to 'hql'.
               (Edward Yoon via Stack)
   HADOOP-2139 (phase 2) Make region server more event driven
   HADOOP-2289 Useless efforts of looking for the non-existant table in select
               command.
               (Edward Yoon via Stack)
   HADOOP-2257 Show a total of all requests and regions on the web ui
               (Paul Saab via Stack)
   HADOOP-2261 HTable.abort no longer throws exception if there is no active update.
   HADOOP-2287 Make hbase unit tests take less time to complete.
   HADOOP-2262 Retry n times instead of n**2 times.
   HADOOP-1608 Relational Algrebra Operators
               (Edward Yoon via Stack)
   HADOOP-2198 HTable should have method to return table metadata
   HADOOP-2296 hbase shell: phantom columns show up from select command
   HADOOP-2297 System.exit() Handling in hbase shell jar command
               (Edward Yoon via Stack)
   HADOOP-2224 Add HTable.getRow(ROW, ts)
               (Bryan Duxbury via Stack)
   HADOOP-2339 Delete command with no WHERE clause
               (Edward Yoon via Stack)
   HADOOP-2299 Support inclusive scans (Bryan Duxbury via Stack)
   HADOOP-2333 Client side retries happen at the wrong level
   HADOOP-2357 Compaction cleanup; less deleting + prevent possible file leaks
   HADOOP-2392 TestRegionServerExit has new failure mode since HADOOP-2338
   HADOOP-2370 Allow column families with an unlimited number of versions
               (Edward Yoon via Stack)
   HADOOP-2047 Add an '--master=X' and '--html' command-line parameters to shell
               (Edward Yoon via Stack)
   HADOOP-2351 If select command returns no result, it doesn't need to show the
               header information (Edward Yoon via Stack)
   HADOOP-2285 Add being able to shutdown regionservers (Dennis Kubes via Stack)
   HADOOP-2458 HStoreFile.writeSplitInfo should just call 
               HStoreFile.Reference.write
   HADOOP-2471 Add reading/writing MapFile to PerformanceEvaluation suite
   HADOOP-2522 Separate MapFile benchmark from PerformanceEvaluation
               (Tom White via Stack)
   HADOOP-2502 Insert/Select timestamp, Timestamp data type in HQL
               (Edward Yoon via Stack)
   HADOOP-2450 Show version (and svn revision) in hbase web ui
   HADOOP-2472 Range selection using filter (Edward Yoon via Stack)
   HADOOP-2548 Make TableMap and TableReduce generic
               (Frederik Hedberg via Stack)
   HADOOP-2557 Shell count function (Edward Yoon via Stack)
   HADOOP-2589 Change an classes/package name from Shell to hql
               (Edward Yoon via Stack)
   HADOOP-2545 hbase rest server should be started with hbase-daemon.sh
   HADOOP-2525 Same 2 lines repeated 11 million times in HMaster log upon
               HMaster shutdown
   HADOOP-2616 hbase not spliting when the total size of region reaches max
               region size * 1.5
   HADOOP-2643 Make migration tool smarter.
   
Release 0.15.1
Branch 0.15

  INCOMPATIBLE CHANGES
    HADOOP-1931 Hbase scripts take --ARG=ARG_VALUE when should be like hadoop
                and do ---ARG ARG_VALUE

  NEW FEATURES
    HADOOP-1768 FS command using Hadoop FsShell operations
                (Edward Yoon via Stack)
    HADOOP-1784 Delete: Fix scanners and gets so they work properly in presence
                of deletes. Added a deleteAll to remove all cells equal to or
                older than passed timestamp.  Fixed compaction so deleted cells
                do not make it out into compacted output.  Ensure also that
                versions > column max are dropped compacting.
    HADOOP-1720 Addition of HQL (Hbase Query Language) support in Hbase Shell.
                The old shell syntax has been replaced by HQL, a small SQL-like
                set of operators, for creating, altering, dropping, inserting,
                deleting, and selecting, etc., data in hbase.
                (Inchul Song and Edward Yoon via Stack)
    HADOOP-1913 Build a Lucene index on an HBase table
                (Ning Li via Stack)
    HADOOP-1957 Web UI with report on cluster state and basic browsing of tables

  OPTIMIZATIONS

  BUG FIXES
    HADOOP-1527 Region server won't start because logdir exists
    HADOOP-1723 If master asks region server to shut down, by-pass return of
                shutdown message
    HADOOP-1729 Recent renaming or META tables breaks hbase shell
    HADOOP-1730 unexpected null value causes META scanner to exit (silently)
    HADOOP-1747 On a cluster, on restart, regions multiply assigned
    HADOOP-1776 Fix for sporadic compaction failures closing and moving
                compaction result
    HADOOP-1780 Regions are still being doubly assigned
    HADOOP-1797 Fix NPEs in MetaScanner constructor
    HADOOP-1799 Incorrect classpath in binary version of Hadoop
    HADOOP-1805 Region server hang on exit
    HADOOP-1785 TableInputFormat.TableRecordReader.next has a bug
                (Ning Li via Stack)
    HADOOP-1800 output should default utf8 encoding
    HADOOP-1801 When hdfs is yanked out from under hbase, hbase should go down gracefully
    HADOOP-1813 OOME makes zombie of region server
    HADOOP-1814	TestCleanRegionServerExit fails too often on Hudson
    HADOOP-1820 Regionserver creates hlogs without bound
                (reverted 2007/09/25) (Fixed 2007/09/30)
    HADOOP-1821 Replace all String.getBytes() with String.getBytes("UTF-8")
    HADOOP-1832 listTables() returns duplicate tables
    HADOOP-1834 Scanners ignore timestamp passed on creation
    HADOOP-1847 Many HBase tests do not fail well.
    HADOOP-1847 Many HBase tests do not fail well. (phase 2)
    HADOOP-1870 Once file system failure has been detected, don't check it again
                and get on with shutting down the hbase cluster.
    HADOOP-1888 NullPointerException in HMemcacheScanner (reprise)
    HADOOP-1903 Possible data loss if Exception happens between snapshot and
                flush to disk.
    HADOOP-1920 Wrapper scripts broken when hadoop in one location and hbase in
                another
    HADOOP-1923, HADOOP-1924 a) tests fail sporadically because set up and tear
                 down is inconsistent b) TestDFSAbort failed in nightly #242
    HADOOP-1929 Add hbase-default.xml to hbase jar
    HADOOP-1941 StopRowFilter throws NPE when passed null row
    HADOOP-1966 Make HBase unit tests more reliable in the Hudson environment.
    HADOOP-1975 HBase tests failing with java.lang.NumberFormatException
    HADOOP-1990 Regression test instability affects nightly and patch builds
    HADOOP-1996 TestHStoreFile fails on windows if run multiple times
    HADOOP-1937 When the master times out a region server's lease, it is too 
                aggressive in reclaiming the server's log.
    HADOOP-2004 webapp hql formatting bugs 
    HADOOP_2011 Make hbase daemon scripts take args in same order as hadoop
                daemon scripts
    HADOOP-2017 TestRegionServerAbort failure in patch build #903 and
                nightly #266
    HADOOP-2029 TestLogRolling fails too often in patch and nightlies
    HADOOP-2038 TestCleanRegionExit failed in patch build #927

  IMPROVEMENTS
    HADOOP-1737 Make HColumnDescriptor data publically members settable
    HADOOP-1746 Clean up findbugs warnings
    HADOOP-1757 Bloomfilters: single argument constructor, use enum for bloom
                filter types
    HADOOP-1760 Use new MapWritable and SortedMapWritable classes from
                org.apache.hadoop.io
    HADOOP-1793 (Phase 1) Remove TestHClient (Phase2) remove HClient.
    HADOOP-1794 Remove deprecated APIs
    HADOOP-1802 Startup scripts should wait until hdfs as cleared 'safe mode'
    HADOOP-1833 bin/stop_hbase.sh returns before it completes
                (Izaak Rubin via Stack) 
    HADOOP-1835 Updated Documentation for HBase setup/installation
                (Izaak Rubin via Stack)
    HADOOP-1868 Make default configuration more responsive
    HADOOP-1884 Remove useless debugging log messages from hbase.mapred
    HADOOP-1856 Add Jar command to hbase shell using Hadoop RunJar util
                (Edward Yoon via Stack)
    HADOOP-1928 Have master pass the regionserver the filesystem to use
    HADOOP-1789 Output formatting
    HADOOP-1960 If a region server cannot talk to the master before its lease
                times out, it should shut itself down
    HADOOP-2035 Add logo to webapps


Below are the list of changes before 2007-08-18

  1. HADOOP-1384. HBase omnibus patch. (jimk, Vuk Ercegovac, and Michael Stack)
  2. HADOOP-1402. Fix javadoc warnings in hbase contrib. (Michael Stack)
  3. HADOOP-1404. HBase command-line shutdown failing (Michael Stack)
  4. HADOOP-1397. Replace custom hbase locking with 
     java.util.concurrent.locks.ReentrantLock (Michael Stack)
  5. HADOOP-1403. HBase reliability - make master and region server more fault
     tolerant.
  6. HADOOP-1418. HBase miscellaneous: unit test for HClient, client to do
     'Performance Evaluation', etc.
  7. HADOOP-1420, HADOOP-1423. Findbugs changes, remove reference to removed 
     class HLocking.
  8. HADOOP-1424. TestHBaseCluster fails with IllegalMonitorStateException. Fix
     regression introduced by HADOOP-1397.
  9. HADOOP-1426. Make hbase scripts executable + add test classes to CLASSPATH.
 10. HADOOP-1430. HBase shutdown leaves regionservers up.
 11. HADOOP-1392. Part1: includes create/delete table; enable/disable table;
     add/remove column.
 12. HADOOP-1392. Part2: includes table compaction by merging adjacent regions
     that have shrunk in size.
 13. HADOOP-1445 Support updates across region splits and compactions
 14. HADOOP-1460 On shutdown IOException with complaint 'Cannot cancel lease
     that is not held'
 15. HADOOP-1421 Failover detection, split log files.
     For the files modified, also clean up javadoc, class, field and method 
     visibility (HADOOP-1466)
 16. HADOOP-1479 Fix NPE in HStore#get if store file only has keys < passed key.
 17. HADOOP-1476 Distributed version of 'Performance Evaluation' script
 18. HADOOP-1469 Asychronous table creation
 19. HADOOP-1415 Integrate BSD licensed bloom filter implementation.
 20. HADOOP-1465 Add cluster stop/start scripts for hbase
 21. HADOOP-1415 Provide configurable per-column bloom filters - part 2.
 22. HADOOP-1498. Replace boxed types with primitives in many places.
 23. HADOOP-1509.  Made methods/inner classes in HRegionServer and HClient protected
     instead of private for easier extension. Also made HRegion and HRegionInfo public too.
     Added an hbase-default.xml property for specifying what HRegionInterface extension to use
     for proxy server connection. (James Kennedy via Jim Kellerman)
 24. HADOOP-1534. [hbase] Memcache scanner fails if start key not present
 25. HADOOP-1537. Catch exceptions in testCleanRegionServerExit so we can see
     what is failing.
 26. HADOOP-1543 [hbase] Add HClient.tableExists
 27. HADOOP-1519 [hbase] map/reduce interface for HBase.  (Vuk Ercegovac and
     Jim Kellerman)
 28. HADOOP-1523 Hung region server waiting on write locks 
 29. HADOOP-1560 NPE in MiniHBaseCluster on Windows
 30. HADOOP-1531 Add RowFilter to HRegion.HScanner
     Adds a row filtering interface and two implemenentations: A page scanner,
     and a regex row/column-data matcher. (James Kennedy via Stack)
 31. HADOOP-1566 Key-making utility
 32. HADOOP-1415 Provide configurable per-column bloom filters. 
     HADOOP-1466 Clean up visibility and javadoc issues in HBase.
 33. HADOOP-1538 Provide capability for client specified time stamps in HBase
     HADOOP-1466 Clean up visibility and javadoc issues in HBase.
 34. HADOOP-1589 Exception handling in HBase is broken over client server connections
 35. HADOOP-1375 a simple parser for hbase (Edward Yoon via Stack)
 36. HADOOP-1600 Update license in HBase code
 37. HADOOP-1589 Exception handling in HBase is broken over client server
 38. HADOOP-1574 Concurrent creates of a table named 'X' all succeed
 39. HADOOP-1581 Un-openable tablename bug
 40. HADOOP-1607 [shell] Clear screen command (Edward Yoon via Stack)
 41. HADOOP-1614 [hbase] HClient does not protect itself from simultaneous updates
 42. HADOOP-1468 Add HBase batch update to reduce RPC overhead
 43. HADOOP-1616 Sporadic TestTable failures
 44. HADOOP-1615 Replacing thread notification-based queue with 
     java.util.concurrent.BlockingQueue in HMaster, HRegionServer
 45. HADOOP-1606 Updated implementation of RowFilterSet, RowFilterInterface
     (Izaak Rubin via Stack)
 46. HADOOP-1579 Add new WhileMatchRowFilter and StopRowFilter filters
    (Izaak Rubin via Stack)
 47. HADOOP-1637 Fix to HScanner to Support Filters, Add Filter Tests to
     TestScanner2 (Izaak Rubin via Stack)
 48. HADOOP-1516 HClient fails to readjust when ROOT or META redeployed on new
     region server
 49. HADOOP-1646 RegionServer OOME's under sustained, substantial loading by
     10 concurrent clients
 50. HADOOP-1468 Add HBase batch update to reduce RPC overhead (restrict batches
     to a single row at a time)
 51. HADOOP-1528 HClient for multiple tables (phase 1) (James Kennedy & JimK)
 52. HADOOP-1528 HClient for multiple tables (phase 2) all HBase client side code
     (except TestHClient and HBaseShell) have been converted to use the new client
     side objects (HTable/HBaseAdmin/HConnection) instead of HClient.
 53. HADOOP-1528 HClient for multiple tables - expose close table function
 54. HADOOP-1466 Clean up warnings, visibility and javadoc issues in HBase.
 55. HADOOP-1662 Make region splits faster
 56. HADOOP-1678 On region split, master should designate which host should 
     serve daughter splits. Phase 1: Master balances load for new regions and
     when a region server fails.
 57. HADOOP-1678 On region split, master should designate which host should 
     serve daughter splits. Phase 2: Master assigns children of split region
     instead of HRegionServer serving both children.
 58. HADOOP-1710 All updates should be batch updates
 59. HADOOP-1711 HTable API should use interfaces instead of concrete classes as
     method parameters and return values
 60. HADOOP-1644 Compactions should not block updates
 60. HADOOP-1672 HBase Shell should use new client classes
     (Edward Yoon via Stack).
 61. HADOOP-1709 Make HRegionInterface more like that of HTable
     HADOOP-1725 Client find of table regions should not include offlined, split parents