Documents

1719282 - POS TLOG Table Partitioning Information

Description
SAP Note     1719282 - POS TLOG Table Partitioning Information Version   6     Validity: 20.09.2013 - active     Language   English (Master) Header Data Released On Release Status Component 20.12.2013 14:35:03 Released for Customer BW-BCT-ISR-PIP Point of Sales Inbound Processing Engine Other Components CA-RT-CAR-PIP SAP Customer Activity Repository POS DM & Sales Audit Priority Category Recommendations / Additional Info Performance Symptom If you have installed one of the following: l
Categories
Published
of 4
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Related Documents
Share
Transcript
  SAP Note   Header Data Symptom  If you have installed one of the following: l SAP POS DM 1.0 on SAP NetWeaver BW powered by SAP HANA, or l SAP Customer Activity Repository 1.0, we recommend that you partition the point-of-sale (POS) transaction log (TLOG) tables, as they can contain extremely large volumes of transactional data. Other Terms Reason and Prerequisites Solution  This note provides information on how to partition the following TLOG tables: l /POSDW/TLOGF: Stores transaction information received from the POS on the SAP HANA database l /POSDW/TLOGF_EXT: Stores customer extensions on the SAP HANA database General Remarks  This section provides sample SQL statements. The schema name used here is SAP<sid>. When at the customer site, you must replace this value with the schema name defined for the customer. The following commands can only be executed by a user who has the appropriate usage privileges. This section assumes that the user executing the commands has database system user privileges. If we exceed 2 to the power of 31 entries, commands entered in a native SQL editor console will fail. We recommend that you occasionally check the amount of free log space in the system - especially during mass insertion scenarios such as migration after install. There is an SQL command that allows you to free up log space. However, you can only execute the command if a backup was recently executed. To free up log file space after a backup, enter the following SQL command: ALTER SYSTEM savepoint;   ALTER SYSTEM reclaim LOG;  If you partition a table, you must supply a special partition named OTHERS. This partition will handle all entries that do not fit into a specified range. After the initial partition creation, you can delete the OTHERS partition. However any data insertion that does not fall into a specified range would lead to a hard database dump. We recommend you leave the OTHERS partition in the database in the case of unforeseen situations where no prepared range partition is defined. If you want to delete the OTHERS partition, execute the following command in Studio. ..........................   ; Drop others partition   ALTER TABLE SAP<sid> . /POSDW/TLOGF DROP PARTITION OTHERS;   Before Partitioning  Starting with revision 53 of SAP HANA, bulk operations have been optimized on the database so that secondary keys are no longer necessary (as stated in earlier versions of this note.) Precondition for that is also that the primary keys on the /POSDW/TLOGF and /POSDW/TLOGF_EXT tables are keep intact and are not dropped on the SAP HANA database. There are no additional steps to be taken before partitioning. 1719282 - POS TLOG Table Partitioning Information   Version  6 Validity:  20.09.2013 - active Language  English (Master) Released On  20.12.2013 14:35:03 Release Status  Released for Customer Component  BW-BCT-ISR-PIP Point of Sales Inbound Processing Engine CA-RT-CAR-PIP SAP Customer Activity Repository POS DM & Sales Audit Priority  Recommendations / Additional Info Category  Performance Other Components  Database Sizing  Refer to SAP service Marketplace at service.sap.com/sizing for further information on database sizing. Partitioning Scheme Definition  The partitioning scheme applied to /POSDW/TLOGF and /POSDW/TLOGF_EXT tables has several objectives. To distribute the data load between several active nodes of the SAP HANA database, a HASH partitioning on the first partitioning level is used. This assures an even distribution of data to all available nodes. It is important to base partitioning on fields that are used in the most important queries that are applied during runtime. Thus, the data is partitioned based on the CLIENT, RETAILSTOREID and BUSINESSDAYDATE columns. You may leave out the BUSINESSDAYDATE column from the HASH definition if the specification of the RETAILSTOREID as it is implemented in the customer scenario guarantees an even distribution of data. If your implementation supports only a few stores, or if the size of the actual stores differs significantly, we recommend including the BUSINESSDAYDATE column. The second partitioning level is to remain within the physical bounds that apply to table size with respect to the number of entries, the physical bounds of the overall number of partitions, and a heuristic to keep low the overhead for partitioning results consolidation. We recommend that you only use partitioning if a table or partition has more than 250 million entries. The maximum number of entries for a table or partition should be 1,000 million in order to avoid the physical limit of 2 to the power of 31 entries. To determine the number of partitions required for the /POSDW/TLOGF and /POSDW/TLOGF_EXT tables, consider the following inputs: 1. D: Number of days Determine the time range that is to be kept in the system to provide enough data for proper forecasting or historical comparison. This will usually vary between one to three years. 2. T: Number of transactions generated per day 3. E: Average number of table entries Determine the average number of table entries required for storing sales transactions. This is dominated by the average number of line items that are to be expected for a typical sales transaction. Given this number, you can estimate the number of table rows approximately as 15+5*N, where N is the number of line items per sales transaction. If you are using customer extensions to attach additional information to POS transaction data, you have to take them into account as well. Customer extensions can be either included in the /POSDW/TLOGF table or they can be stored separately in the /POSDW/TLOGF_EXT, depending on your Customizing settings. a) Case A: When storing customer extensions in /POSDW/TLOGF, each extension field adds another row to each transaction record. For counting these extension rows apply 15+5*N*X for the value of E, where X is the number of extension fields per transaction. If you have existing TLOG data from earlier productive runs of POS DM it is most reliable to count the corresponding average row counts from your existing data. If this is not possible, you have to apply the formula above for estimation of the E value. b) Case B: When storing customer extensions in /POSDW/TLOGF_EXT, regular transaction data is completely separated from transaction extension data. In this case you have to find out whether the the maximum row count is in table /POSDW/TLOGF or table/POSDW/TLOGF_EXT. To find this out, you have to determine wether there are more regular line items or more extensions fields per transaction. E is MAX(15+5*N,X) where X is the number of extension fields per transaction. 4. Calculate the expected total table size as D*T*E. 5. Select an appropriate time range D so that D*T*E falls between 250 million and 1,000 million. Choose the date span of D either as monthly, quarterly or yearly. a) Case A: When running on a single-node database system you're done with the found value for D. b) Case B: When running on a multiple-node database system (also known as scale-out system), you have to correct the computation for the number of effective nodes that you are planning to utilize for POS DM. This situation may be the case if the overall data volume does not fit any more on a single database node because of physical memory limitations of that node. Let's assume that you are planning to supply M nodes that carry POS data. Then you have to select the appropriate time range D so that (D*T*E)/M falls between 250 million and 1,000 million. So a selected time range D can contain more data than can physically fit into a single range partition, because the data is distributed amongst the M available nodes. Examples  Example 1 (single node): Assume that we have a customer scenario with the following data: l Two years of historical data: D = 2 * 365 l Stores that produce 1,500,000 transactions per day: T = 1,500,000 l Twelve (12) line items per sales transaction: E = 15+5*12=75 The expected total /POSDW/TLOGF table size would be 730*1.500.000*75 = 82.125M entries.   The minimum number of partitions to stay below 1,000 entries would be 82. The maximum number of partitions to start partitioning with at least 250 million entries would be 329. If we chose weekly partitions, we would end up with 104 partitions with each containing approximately 790 million entries. Example 2 (multiple node): Assume that we have a customer scenario with the following data: l Three years of historical data: D = 3 * 365 = 1095 l Stores generate 5,000,000 transactions per day: T = 5,000,000 l 15 line items per sales transaction: E = 15+5*15 = 90 l In average 50 customer extension fields per transaction: X = 50 l 16 nodes planned (1 master node, 1 fail-over node, 14 regular nodes): M = 14 Because there is a significant amount of extension fields, Customizing is set to store extensions in the separate table /POSDW/TLOGF_EXT. Looking at the expected row count per transaction /POSDW/TLOGF still holds more rows than /POSDW/TLOGF_EXT (90 > 50). So for sizing consideration of the range partitioning it is sufficient to look at /POSDW/TLOGF. The expected total /POSDW/TLOGF table size would be 1095*5.000.000*90 = 492.750M entries. Because we run with 14 database nodes, the data for each node is about 35.196M entries. Going with yearly partitions would bring down the row count per partition and node to 11.732M, which is too large (2B max). Going with quarterly partitions would bring down the row count per partition and node to 2.933M, which still is too large (2B max). Going with monthly partitions brings down the row count per partition and node to 978M, which neatly fits to the given bounds (>250M, <2B). Initial Partitioning  The initial partitioning consist of the creation of the partitions for /POSDW/TLOGF and /POSDW/TLOGF_EXT tables and the relocation of the first level partitions onto their corresponding processing nodes if running on a distributed landscape. Partitioning  Once you have estimated the total size of the TLOG tables, you can begin the initial partitioning of the tables. All TLOG tables to be partitioned (/POSDW/TLOGF, /POSDW/TLOGF_EXT) have a common set of primary keys. You have to use this very same set of primary keys for the first partitioning level of all tables. You should also apply the very same secondary partitioning scheme for all to be partitioned TLOG tables. You can partition empty or populated tables. However, partitioning populated tables is time consuming due to the number of indices that have to be created. You should also be aware that a modification of a partitioning on the first partitioning level can only take place if the complete data set to be partitioned fits onto a single database node. Therefore, we recommend that you partition the tables before a large amount of data is added. The following command is executed in SAP HANA Studio to partition a table. Please note that the ellipsis would have to be expanded to the full set of weekly date ranges for a two-year period. In this example, the MANDT, RETAILSTOREID, and BUSINESSDAYDATE fields are used to define a HASH value to distribute the data evenly among the available nodes. The range partitioning is defined over weekly periods of field BUSINESSDAYDATE starting at 20000101. Also note that the upper bound of each date range is interpreted as being exclusive. ; Create weekly partition with others partition   alter table SAP<sid> . /POSDW/TLOGF partition by HASH ( mandt,retailstoreid, businessdaydate ) PARTITIONS GET_NUM_SERVERS(), RANGE (businessdaydate ) ( PARTITION 20000101 <= VALUES < 20000108,   PARTITION 20000108 <= VALUES < 20000115,   PARTITION 20000115 <= VALUES < 20000122,   PARTITION 20000122 <= VALUES < 20000129,   #   PARTITION 20011225 <= VALUES < 20020101,   PARTITION OTHERS);   ; Should produce a message in Hana Studio like:   Statement 'ALTER TABLE SAP<sid> . /POSDW/TLOGF partition by HASH ( mandt, retailstoreid, businessdaydate ) PARTITIONS ...' successfully executed in 559 ms 754 µs (server processing time: 452 ms 983 µ s) - Rows Affected: 0 ; You can verify the results using following statement:   SELECT * FROM m_cs_tables   WHERE schema_name = 'SAP<sid> ' AND table_name = '/POSDW/TLOGF';  You can also verify the result in SAP HANA Studio: 1. In Studio, choose the Find Table function. 2. Specify /POSDW/TLOGF as the table name.    3. Choose the Show Definition option. 4. Choose the Runtime Information tab in the results window to view the partitioning information. Relocating Partitions  If you are running on a multiple-node system you also have to explicitly move the generated partitions to the respective nodes. To move all TLOG-related partitions of a landscape with 4 nodes onto its processing node, the command to accomplish this is similar to the following: (Let node 1 be the master node that does not carry any data. Let the nodes HANASRV2-HANASRV4 be the worker nodes, the system name be '<SID>', the system id be '11' and the partitions to be moved be '1' to '3'.) alter table SAP<SID> . /POSDW/TLOGF move partition 1 to HANASRV2:31103 physical; alter table SAP<SID> . /POSDW/TLOGF move partition 2 to HANASRV3:31103 physical; alter table SAP<SID> . /POSDW/TLOGF move partition 3 to HANASRV4:31103 physical; alter table SAP<SID> . /POSDW/TLOGF_EXT move partition 1 to HANASRV2:31103 physical; alter table SAP<SID> . /POSDW/TLOGF_EXT move partition 2 to HANASRV3:31103 physical; alter table SAP<SID> . /POSDW/TLOGF_EXT move partition 3 to HANASRV4:31103 physical; Please note that the port numbers start with '3' followed by the system id and end with '03'. Validity References This document refers to: SAP Notes   This document is referenced by: SAP Notes (2)  Software Component From Rel. To Rel. And Subsequent RTLPOSDM 100_700 100_700   100_730 100_730   100_731 100_731   1852602 Storing POS Transaction extensions in a separate table 1852602 Storing POS Transaction extensions in a separate table 2014446 Maintain POS Transaction tables Level 2 range partitions 
Search
Similar documents
View more...
Tags
Related Search
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks