Class that reads CLI output and parses into tabular format. Marrying cli command with additional meta data in order to select the appropriate !TextFSM template for parsing the command output.
Uses an index file that contains a table for mapping command strings to templates.
An index table corresponding to the templates supplied in the examples directory looks like the following:
# First line is the header fields for columns and is mandatory.
# Regular expressions are supported in all fields except the first.
# Last field supports variable length command completion.
# abc[xyz](xyz) is expanded to abc(x(y(z)?)?)?, regexp inside [[]] is not supported
#
Template, Hostname, Vendor, Command
cisco_bgp_summary_template, .*, Cisco, sh[ow](ow) ip bg[p](p) su[mmary](mmary)
cisco_version_template, .*, Cisco, sh[ow](ow) ve[rsion](rsion)
f10_ip_bgp_summary_template, .*, Force10, sh[ow](ow) ip bg[p](p) sum[mary](mary)
f10_version_template, .*, Force10, sh[ow](ow) ve[rsion](rsion)
juniper_bgp_summary_template, .*, Juniper, sh[ow](ow) bg[p](p) su[mmary](mmary)
juniper_version_template, .*, Juniper, sh[ow](ow) ve[rsion](rsion)
unix_ifcfg_template, hostname[abc].*, .*, ifconfigThe templates are defined in the first column and the CLI command is defined in the last column. These columns are special and their location is fixed. There is then an optional number of columns in between that can be used to match meta data.
In our example there have been defined two optional columns for the attributes (meta data) Hostname and Vendor.
import clitable
# Read in some example command outputs.
raw_cmd_data1 = open('examples/cisco_version_example').read()
raw_cmd_data2 = open('examples/f10_version_example').read()
# Initialise CliTable with the content of the 'index' file.
cli_table = clitable.CliTable('index', 'examples')
# Firstly we will use attributes to match a 'show version' command on a Cisco device.
attributes = {'Command': 'show version' , 'Vendor': 'Cisco'}
cli_table.ParseCmd(raw_cmd_data1, attributes)
print cli_table
>>> Model, Memory, ConfigRegister, Uptime, Version, ReloadReason, ReloadTime, ImageFile
>>> WS-C4948-10GE, 262144K, 0x2102, 3 days, 13 hours, 53 minutes, 12.2(31)SGA1, reload, 05:09:09 PDT Wed Apr 2 2008, bootflash:cat4500-entservicesk9-mz.122-31.SGA1.bin
# This time, the same command but on a Force10.
attributes['Vendor'] = 'Force10'
cli_table.ParseCmd(raw_cmd_data2, attributes)
print cli_table
>>> Chassis, Model, Software, Image
>>> E1200, E1200, 7.7.1.1, flash://FTOS-EF-7.7.1.1.binThe command column in the index file uses the special syntax for variable length command completion.
# Incomplete command strings are supported.
attributes['Command'] = 'sh vers'
cli_table.ParseCmd(raw_cmd_data2, attributes)
print cli_table
>>> Chassis, Model, Software, Image
>>> E1200, E1200, 7.7.1.1, flash://FTOS-EF-7.7.1.1.binCliTables with identical column names can be added together.
ct = cli_table + cli_table
print ct
>>> Chassis, Model, Software, Image
>>> E1200, E1200, 7.7.1.1, flash://FTOS-EF-7.7.1.1.bin
>>> E1200, E1200, 7.7.1.1, flash://FTOS-EF-7.7.1.1.binCliTables are built upon a TextTable object, and can use all the features in that class.
ct.Append(('E1200', 'E1200', '7.7.2.2', 'flash://FTOS-EF-7.7.2.2.bin'))
print ct.FormattedTable()
>>> Chassis Model Software Image
>>> =======================================================
>>> E1200 E1200 7.7.1.1 flash://FTOS-EF-7.7.1.1.bin
>>> E1200 E1200 7.7.1.1 flash://FTOS-EF-7.7.1.1.bin
>>> E1200 E1200 7.7.2.2 flash://FTOS-EF-7.7.2.2.bin This codelab is designed to help users of the TextFSM python module in writing new templates. This flexible generic text parser is primarily used for parsing the output of router CLI commands in Python. By writing relatively simple regex based text files, a multitude of command outputs can be parsed with minimal code.
This codelab starts with the output of a simple router command, and will guide you in the steps required to successfully write a template and use it in a simple Python script.
You should already have a basic understanding in Python and have read the TextTableFSM Howto page, and have it open for reference.
For the first example, we will parse simple linear (non-row based) text - the output of a Cisco 'show clock':
18:42:41.321 PST Sun Feb 8 2009Only the actual date/time is going to be parsed - we don't parse the prompts. The information that we'd like to extract here is:
Each of these bits of information will be extracted into its own FSM 'Value', which will make each one individually available to a Python script using it. For this example, we will intentionally not try to match the weekday. Thus the first thing we need to do is populate 'Value' lines into our template.
Open a new file, called cisco_clock_template, and put the following lines into it:
Value Year (\d+)
Value MonthDay (\d+)
Value Month (\w+)
Value Timezone (\S+)
Value Time (..:..:..)
These are our Value definitions. On each line, the first word is the keyword 'Value', and tells the FSM that the line defines a column in a row. The next word is the name of the value - this can be any alphanumeric word. Lastly we have a regex containing at least one paren-wrapped expression. These are sub-expressions containing a regex to match exactly the value required. For example, the (\d+) will match the numeric digit sequence denoting a year.
Insert a blank line after the 'Time' value definition. This denotes the end of the 'Value' definitions. After this, we start defining states of our FSM. The state 'Start' is always required, and the FSM will begin here when first parsing text. So the next line of the FSM has the single word 'Start':
Value Year (\d+)
Value MonthDay (\d+)
Value Month (\w+)
Value Timezone (\S+)
Value Time (..:..:..)
StartFollowing the state label, we insert one or more rules. When the state machine enters a new state, it takes its current line of input, and attempts to match it against each rule in turn. Once matched, any 'Value's are substituted and optional actions are taken. This template will have only one rule, so complete the template as follows, and save it as cisco_clock_template:
Value Year (\d+)
Value MonthDay (\d+)
Value Month (\w+)
Value Timezone (\S+)
Value Time (..:..:..)
Start
^${Time}.* ${Timezone} \w+ ${Month} ${MonthDay} ${Year} -> RecordInternally, the rule(s) will be expanded, such that any reference to a Value will be replaced by the regex specified in the Value. The single rule in this template will be expanded internally to this regex:
^(..:..:..).* (\S+) \w+ (\w+) (\d+) (\d+)When the single line of input (18:42:41.321 PST Sun Feb 8 2009) is applied to it, the regex will match, and each of the regex match groups will contain the following values:
| Group | Value |
|---|---|
| 1 | 18:42:41 |
| 2 | PST |
| 3 | Feb |
| 4 | 8 |
| 5 | 2009 |
Because of the successful match, each of the Values associated with the group will be assigned the matching text. This will fill the one (and only) row.
We also have an associated action Record. This instructs the FSM to insert the row into the final table. There is also an implicit 'Next' action (see TextTableFSM that tells the FSM to read the next line of text in. As there is no further text (we only have one line), the FSM completes and returns just the single matched row.
We will try the FSM on some input text:
$ cat > router_output.txt
18:42:41.321 PST Sun Feb 8 2009
$ ./parser.py cisco_clock_template router_output.txt
FSM Template:
Value Year (\d+)
Value MonthDay (\d+)
Value Month (\w+)
Value Timezone (\S+)
Value Time (..:..:..)
Start
^${Time}.* ${Timezone} \w+ ${Month} ${MonthDay} ${Year} -> RecordFSM Table:
['Year', 'MonthDay', 'Month', 'Timezone', 'Time']
['2009', '8', 'Feb', 'PST', '18:42:41']Success! The first section of the output shows the parsed FSM, and the last section shows the final table. The first row of this table is the header, showing the columns (or 'Value's) that we've defined. The next line is the row of values that were matched.
We'll now look at something slightly more complicated - matching values from multiple lines of text. For this, we will attempt to parse the output of a Cisco 'show version' command. It looks like this:
Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500-ENTSERVICESK9-M), Version 12.2(31)SGA1, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Fri 26-Jan-07 14:28 by kellythw
Image text-base: 0x10000000, data-base: 0x118AD800
ROM: 12.2(31r)SGA
Pod Revision 0, Force Revision 34, Gill Revision 20
router.abc uptime is 11 weeks, 4 days, 20 hours, 26 minutes
System returned to ROM by reload
System restarted at 22:49:40 PST Tue Nov 18 2008
System image file is "bootflash:cat4500-entservicesk9-mz.122-31.SGA1.bin"
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
export@cisco.com.
cisco WS-C4948-10GE (MPC8540) processor (revision 5) with 262144K bytes of memory.
Processor board ID FOX111700ZN
MPC8540 CPU at 667Mhz, Fixed Module
Last reset from Reload
2 Virtual Ethernet interfaces
48 Gigabit Ethernet interfaces
2 Ten Gigabit Ethernet interfaces
511K bytes of non-volatile configuration memory.
Configuration register is 0x2102We will be extracting several bits of information from this:
This gives us four columns in our table (which will end up as a single row). To start, we will thus define our 'Value' statements. Start with a new file, called 'cisco_version_template' and put the following into it:
Value Version ([^ ,]+)
Value Uptime (.*)
Value ConfigRegister (\w+)
Value ResetReason (.*)
Start
^Cisco IOS .*Version ${Version},
^.*uptime is ${Uptime}
^Last reset from ${ResetReason}
^Configuration register is ${ConfigRegister} -> RecordThis is a little more complicated that the first example. Most importantly, the template only matches some of the input lines. For each line of input, each rule in the state is checked. If there is no match with a rule, the FSM simply moves to the next rule. Once all rules have been checked (in this case, after the 'Configuration register' rule), then the next line of input is fetched, and rule are checked from the start of the state again.
So the following will happen here:
The first line of input will be matched, and the 'Version' value is assigned "12.2(31)SGA1". Until the 'uptime is' line, all other lines fail to match and are discarded. The 'uptime' line will match, and 'Uptime' is assigned "11 weeks, 4 days, 20 hours, 26 minutes" Lines are not matched, and thus discarded, until the 'Last reset' line At this point, there is a match and 'ResetReason' value is assigned 'Reload' Several more lines pass until 'ConfigRegister' is assigned '0x2102' This line also triggers a 'Record' action, and the row is saved. After this line, we reach EOF and the FSM terminates. Run it against the FSM, after saving your template and copying the router output to 'router_output.txt':
$ ./parser.py cisco_version_template router_output.txt
FSM Template:
Value Version ([^ ,]+)
Value Uptime (.*)
Value ConfigRegister (\w+)
Value ResetReason (.*)
Start
^Cisco IOS .*Version ${Version},
^.*uptime is ${Uptime}
^Last reset from ${ResetReason}
^Configuration register is ${ConfigRegister} -> Record
FSM Table:
['Version', 'Uptime', 'ConfigRegister', 'ResetReason']
['12.2(31)SGA1', '11 weeks, 4 days, 20 hours, 26 minutes', '0x2102', 'Reload']The previous examples showed the simplest application of the FSM - parsing non-repeating data into a single row. In this example, we will look at a more interesting example - the output of the Juniper command show chassis fpc will be parsed. The raw router output from one device looks like this:
Temp CPU Utilization (%) Memory Utilization (%)
Slot State (C) Total Interrupt DRAM (MB) Heap Buffer
0 Online 24 7 0 256 38 51
1 Online 25 7 0 256 38 51
2 Online 24 3 0 256 37 49
3 Online 23 3 0 256 37 49
4 Empty
5 Empty
6 Empty
7 EmptyThis text is simple to parse - its information is neatly contained in separate rows, and the FSMs data is always presented as rows. This compatibility will ensure a relatively painless template creation. For our application we'll create a single row of data for each FPC defined, but this will be easy as the text is presented as one line per FPC slot.
The information we want to extract is:
This give us five values of interest. Create a new file juniper_fpc_template, and first put in it the following Value definitions:
Value Slot (\d)
Value State (\w+)
Value Temperature (\d+)
Value DRAM (\d+)
Value Buffer (\d+)Next we put in the State definition and associated rules. We may be able to get away with a single rule.
Value Slot (\d)
Value State (\w+)
Value Temperature (\d+)
Value DRAM (\d+)
Value Buffer (\d+)
Start
# We can't use .* for unused placeholders, as greedy matching will break it.
^\s+${Slot}\s+${State}\s+${Temperature}\s+\d+\s+\d+\s+${DRAM}\s+\d+\s+${Buffer} -> RecordCopy the above raw router output to router_output.txt, save the template, and run it:
$ ./parser.py juniper_fpc_template router_output.txt
FSM Template:
Value Slot (\d)
Value State (\w+)
Value Temperature (\d+)
Value DRAM (\d+)
Value Buffer (\d+)
Start
^\s+${Slot}\s+${State}\s+${Temperature}\s+\d+\s+\d+\s+${DRAM}\s+\d+\s+${Buffer} -> Record
FSM Table:
['Slot', 'State', 'Temperature', 'DRAM', 'Buffer']
['0', 'Online', '24', '256', '51']
['1', 'Online', '25', '256', '51']
['2', 'Online', '24', '256', '49']
['3', 'Online', '23', '256', '49']Our template has successfully parsed all of the online FPC slots and extracted the correct values.
But what about empty slots? If we need to also extract information about every slot, then we won't see them here as the empty ones only have the Slot and State columns. The way to deal with this is to create a second rule to catch these lines.
Add one more rule after the existing one, as follows:
Value Slot (\d)
Value State (\w+)
Value Temperature (\d+)
Value DRAM (\d+)
Value Buffer (\d+)
State
# We can't use .* for unused placeholders, as greedy matching will break it.
^\s+${Slot}\s+${State}\s+${Temperature}\s+\d+\s+\d+\s+${DRAM}\s+\d+\s+${Buffer} -> Record
^\s+${Slot}\s+${State} -> RecordNow run the FSM:
$ ./parser.py juniper_fpc_template router_output.txt
FSM Template:
Value Slot (\d)
Value State (\w+)
Value Temperature (\d+)
Value DRAM (\d+)
Value Buffer (\d+)
Start
^\s+${Slot}\s+${State}\s+${Temperature}\s+\d+\s+\d+\s+${DRAM}\s+\d+\s+${Buffer} -> Record
^\s+${Slot}\s+${State} -> Record
FSM Table:
['Slot', 'State', 'Temperature', 'DRAM', 'Buffer']
['0', 'Online', '24', '256', '51']
['1', 'Online', '25', '256', '51']
['2', 'Online', '24', '256', '49']
['3', 'Online', '23', '256', '49']
['4', 'Empty', '', '', '']
['5', 'Empty', '', '', '']
['6', 'Empty', '', '', '']
['7', 'Empty', '', '', '']Success! We've now extracted info about both filled and empty slots.
Now that we can extract a table from show chassis fpc, we have to do an enhancement. Whilst it works with the simple example above, it isn't entirely effective for the more complicated output from a Juniper TX matrix, which looks like this:
lcc0-re0:
--------------------------------------------------------------------------
Temp CPU Utilization (%) Memory Utilization (%)
Slot State (C) Total Interrupt DRAM (MB) Heap Buffer
0 Online 24 8 1 512 16 52
1 Online 23 7 1 256 36 53
2 Online 23 5 1 256 36 49
3 Online 21 7 1 256 36 49
4 Empty
5 Empty
6 Empty
7 Empty
lcc1-re1:
--------------------------------------------------------------------------
Temp CPU Utilization (%) Memory Utilization (%)
Slot State (C) Total Interrupt DRAM (MB) Heap Buffer
0 Online 20 9 1 256 36 50
1 Online 20 13 0 256 36 49
2 Online 21 6 1 256 36 49
3 Online 20 6 0 256 36 49
4 Online 18 5 0 256 35 49
5 Empty
6 Empty
7 EmptyThere are two physical chassis in this node (lcc0-re0 and lcc1-re0), and we have two of each slot number, one per chassis.
If we run this output against the FSM, we will see the following:
FSM Template:
Value Slot (\d)
Value State (\w+)
Value Temperature (\d+)
Value DRAM (\d+)
Value Buffer (\d+)
Start
^\s+${Slot}\s+${State}\s+${Temperature}\s+\d+\s+\d+\s+${DRAM}\s+\d+\s+${Buffer} -> Record
^\s+${Slot}\s+${State} -> Record
FSM Table:
['Slot', 'State', 'Temperature', 'DRAM', 'Buffer']
['0', 'Online', '24', '512', '52']
['1', 'Online', '23', '256', '53']
['2', 'Online', '23', '256', '49']
['3', 'Online', '21', '256', '49']
['4', 'Empty', '', '', '']
['5', 'Empty', '', '', '']
['6', 'Empty', '', '', '']
['7', 'Empty', '', '', '']
['0', 'Online', '20', '256', '50']
['1', 'Online', '20', '256', '49']
['2', 'Online', '21', '256', '49']
['3', 'Online', '20', '256', '49']
['4', 'Online', '18', '256', '49']
['5', 'Empty', '', '', '']
['6', 'Empty', '', '', '']
['7', 'Empty', '', '', '']Unfortunately there's no way to see if the 512Mb FPC is in lcc0 or lcc1, because we have not stored and chassis info. So we can do this by adding another value 'Chassis' which matches ^\S+:. Update the template by adding the new Value and rule:
Value Chassis (\S+)
Value Slot (\d)
Value State (\w+)
Value Temperature (\d+)
Value DRAM (\d+)
Value Buffer (\d+)
Start
^${Chassis}:
^\s+${Slot}\s+${State}\s+${Temperature}\s+\d+\s+\d+\s+${DRAM}\s+\d+\s+${Buffer} -> Record
^\s+${Slot}\s+${State} -> RecordBut when this is run, there is a slight problem:
FSM Template:
Value Chassis (\S+)
Value Slot (\d)
Value State (\w+)
Value Temperature (\d+)
Value DRAM (\d+)
Value Buffer (\d+)
Start
^${Chassis}:
^\s+${Slot}\s+${State}\s+${Temperature}\s+\d+\s+\d+\s+${DRAM}\s+\d+\s+${Buffer} -> Record
^\s+${Slot}\s+${State} -> Record
FSM Table:
['Chassis', 'Slot', 'State', 'Temperature', 'DRAM', 'Buffer']
['lcc0-re0', '0', 'Online', '24', '512', '52']
['', '1', 'Online', '23', '256', '53']
['', '2', 'Online', '23', '256', '49']
['', '3', 'Online', '21', '256', '49']
['', '4', 'Empty', '', '', '']
['', '5', 'Empty', '', '', '']
['', '6', 'Empty', '', '', '']
['', '7', 'Empty', '', '', '']
['lcc1-re1', '0', 'Online', '20', '256', '50']
['', '1', 'Online', '20', '256', '49']
['', '2', 'Online', '21', '256', '49']
['', '3', 'Online', '20', '256', '49']
['', '4', 'Online', '18', '256', '49']
['', '5', 'Empty', '', '', '']
['', '6', 'Empty', '', '', '']
['', '7', 'Empty', '', '', '']The 'Chassis' column is only filled for the first slot in each chassis. Why?
The problem is that each time a 'Record' action is taken, the row is saved, then each Value is cleared. Thus when corresponding rows are filled and saved, there is no longer anything to match the 'Chassis' rule, so this never gets filled until the next chassis specification in the router output.
To fix this, we will add the 'Filldown' option to the Chassis value. This option tells the FSM to retain the value after each 'Record', so that it isnt cleared. Its value will only change if there is either a 'Clearall' action, or if a rule overwrites the value.
Modify the Chassis Value line with a 'Filldown' option, as follows:
Value Filldown Chassis (\S+)Now run the text through the FSM again:
$ ./parser.py juniper_fpc_template router_output.txt
FSM Template:
Value Filldown Chassis (\S+)
Value Slot (\d)
Value State (\w+)
Value Temperature (\d+)
Value DRAM (\d+)
Value Buffer (\d+)
Start
^${Chassis}:
^\s+${Slot}\s+${State}\s+${Temperature}\s+\d+\s+\d+\s+${DRAM}\s+\d+\s+${Buffer} -> Record
^\s+${Slot}\s+${State} -> Record
FSM Table:
['Chassis', 'Slot', 'State', 'Temperature', 'DRAM', 'Buffer']
['lcc0-re0', '0', 'Online', '24', '512', '52']
['lcc0-re0', '1', 'Online', '23', '256', '53']
['lcc0-re0', '2', 'Online', '23', '256', '49']
['lcc0-re0', '3', 'Online', '21', '256', '49']
['lcc0-re0', '4', 'Empty', '', '', '']
['lcc0-re0', '5', 'Empty', '', '', '']
['lcc0-re0', '6', 'Empty', '', '', '']
['lcc0-re0', '7', 'Empty', '', '', '']
['lcc1-re1', '0', 'Online', '20', '256', '50']
['lcc1-re1', '1', 'Online', '20', '256', '49']
['lcc1-re1', '2', 'Online', '21', '256', '49']
['lcc1-re1', '3', 'Online', '20', '256', '49']
['lcc1-re1', '4', 'Online', '18', '256', '49']
['lcc1-re1', '5', 'Empty', '', '', '']
['lcc1-re1', '6', 'Empty', '', '', '']
['lcc1-re1', '7', 'Empty', '', '', '']
['lcc1-re1', '', '', '', '', '']Now we have the Chassis values all 'filled down' in the table. It's almost perfect except for one problem. There is an odd almost-empty row at the bottom of the table. What is it doing there?
That row comes about because of the Filldown option. When the last valid row is saved (lcc-re1, slot 7), the FSM creates a new empty row ready to have values filled in. Normally the FSM will discard empty rows when it terminates, but in this case the 'Filldown' option has populated the 'Chassis' column, so the FSM keeps this non-empty row and saves it when the FSM terminates.
In order to fix this, we will make use of another Value option - Required. This option specifies that the value must have been matched otherwise a row will not be saved. For this template, we'll ensure that 'slot' and 'state' both contain a value, so we'll modify as follows:
Value Required Slot (\d)
Value Required State (\w+)
Now we finally have a correct table:
FSM Template:
Value Filldown Chassis (\S+)
Value Required Slot (\d)
Value Required State (\w+)
Value Temperature (\d+)
Value DRAM (\d+)
Value Buffer (\d+)
Start
^${Chassis}:
^\s+${Slot}\s+${State}\s+${Temperature}\s+\d+\s+\d+\s+${DRAM}\s+\d+\s+${Buffer} -> Record
^\s+${Slot}\s+${State} -> Record
FSM Table:
['Chassis', 'Slot', 'State', 'Temperature', 'DRAM', 'Buffer']
['lcc0-re0', '0', 'Online', '24', '512', '52']
['lcc0-re0', '1', 'Online', '23', '256', '53']
['lcc0-re0', '2', 'Online', '23', '256', '49']
['lcc0-re0', '3', 'Online', '21', '256', '49']
['lcc0-re0', '4', 'Empty', '', '', '']
['lcc0-re0', '5', 'Empty', '', '', '']
['lcc0-re0', '6', 'Empty', '', '', '']
['lcc0-re0', '7', 'Empty', '', '', '']
['lcc1-re1', '0', 'Online', '20', '256', '50']
['lcc1-re1', '1', 'Online', '20', '256', '49']
['lcc1-re1', '2', 'Online', '21', '256', '49']
['lcc1-re1', '3', 'Online', '20', '256', '49']
['lcc1-re1', '4', 'Online', '18', '256', '49']
['lcc1-re1', '5', 'Empty', '', '', '']
['lcc1-re1', '6', 'Empty', '', '', '']
['lcc1-re1', '7', 'Empty', '', '', '']Often it may be necessary to have a particular column contain a list of values. For example when parsing a routing table, there may be multiple next hops per prefix. The below example is such a routing table, simplified for the purposes of this example:
Destination Gateway Dist/Metric Last Change
----------- ------- ----------- -----------
B EX 0.0.0.0/0 via 192.0.2.73 20/100 4w0d
via 192.0.2.201
via 192.0.2.202
via 192.0.2.74
B IN 192.0.2.76/30 via 203.0.113.183 200/100 4w2d
B IN 192.0.2.204/30 via 203.0.113.183 200/100 4w2d
B IN 192.0.2.80/30 via 203.0.113.183 200/100 4w2d
B IN 192.0.2.208/30 via 203.0.113.183 200/100 4w2dFor this example, we will extract the following information:
Copy the above routing table output to a file called 'routes.txt'.
We will build a simple FSM to extract data. Put the below text into a file called 'routes.tmpl':
Value Protocol (\S)
Value Type (\S\S)
Value Required Prefix (\S+)
Value Gateway (\S+)
Value Distance (\d+)
Value Metric (\d+)
Value LastChange (\S+)
Start
^.*----- -> Routes
Routes
^ ${Protocol} ${Type} ${Prefix}\s+via ${Gateway}\s+${Distance}/${Metric}\s+${LastChange} -> RecordNote above that in the 'Start' state we have a special match for a line of dashes. This implicitly discards input text until after the headers are complete. It's not really needed or this case, but is useful to demonstrate for other cases where header data may be confused with actual row data.
Now run the template against the data:
$ ./parser.py route.tmpl routes.txt
FSM Template:
Value Protocol (\S)
Value Type (\S\S)
Value Prefix (\S+)
Value Gateway (\S+)
Value Distance (\d+)
Value Metric (\d+)
Value LastChange (\S+)
Start
^.*----- -> Routes
Routes
^ ${Protocol} ${Type} ${Prefix}\s+via ${Gateway}\s+${Distance}/${Metric}\s+${LastChange} -> Record
FSM Table:
['Protocol', 'Type', 'Prefix', 'Gateway', 'Distance', 'Metric', 'LastChange']
['B', 'EX', '0.0.0.0/0', '192.0.2.73', '20', '100', '4w0d']
['B', 'IN', '192.0.2.76/30', '203.0.113.183', '200', '100', '4w2d']
['B', 'IN', '192.0.2.204/30', '203.0.113.183', '200', '100', '4w2d']
['B', 'IN', '192.0.2.80/30', '203.0.113.183', '200', '100', '4w2d']
['B', 'IN', '192.0.2.208/30', '203.0.113.183', '200', '100', '4w2d']The table is mostly filled...except that we only see the first Gateway for the default prefix. It's easy to see why - the template does not match the lines defining the other gateways. So how to work around this?
The option 'List' is useful here, as it can be used to specify a column as a list of strings, rather than just a single string. So we will add this option to the 'Gateway' value definition:
Value Protocol (\S)
Value Type (\S\S)
Value Required Prefix (\S+)
Value List Gateway (\S+)
Value Distance (\d+)
Value Metric (\d+)
Value LastChange (\S+)
Run it again, and we see that the Gateway columns are now a list, albeit with one entry:
FSM Table:
['Protocol', 'Type', 'Prefix', 'Gateway', 'Distance', 'Metric', 'LastChange']
['B', 'EX', '0.0.0.0/0', ['192.0.2.73'], '20', '100', '4w0d']
['B', 'IN', '192.0.2.76/30', ['203.0.113.183'], '200', '100', '4w2d']
['B', 'IN', '192.0.2.204/30', ['203.0.113.183'], '200', '100', '4w2d']
['B', 'IN', '192.0.2.80/30', ['203.0.113.183'], '200', '100', '4w2d']
['B', 'IN', '192.0.2.208/30', ['203.0.113.183'], '200', '100', '4w2d']To extract the other Gateway entries, we need to do this in two steps:
Not 'Record' until we start the next Prefix entry (in case of additional Gateway lines)., and Parse the other Gateway lines. To take care of the first, we will need to do make two changes. Firstly we remove the existing 'Record' statement. Secondly, we add a statement before the existing one that upon matching a new Prefix entry it 'Record's the existing one, then continues with the current input line:
Routes
^ \S \S\S -> Continue.Record
^ ${Protocol} ${Type} ${Prefix}\s+via ${Gateway}\s+${Distance}/${Metric}\s+${LastChange}The new line does not use value substitutions, as we need to save the previous prefix entry records, not overwrite them. The 'Continue' statement then resumes at the next rule after 'Record'ing the previous data, using the current input line. If we run this now, we see the output is identical to before.
Step two in getting the List to work here is to create a rule to match additional Gateway entries. This will go at the end. Note that we do not Record here - that happens only when the next entry is processed. Remember also that an EOF will also cause the current entry to be 'Record'ed.
Routes
^ \S \S\S -> Continue.Record
^ ${Protocol} ${Type} ${Prefix}\s+via ${Gateway}\s+${Distance}/${Metric}\s+${LastChange}
^\s+via ${Gateway}Now we run it, and note that we now have our list of all Gateway entries for the default Prefix:
FSM Table:
['Protocol', 'Type', 'Prefix', 'Gateway', 'Distance', 'Metric', 'LastChange']
['B', 'EX', '0.0.0.0/0', ['192.0.2.73', '192.0.2.201', '192.0.2.202', '192.0.2.74'], '20', '100', '4w0d']
['B', 'IN', '192.0.2.76/30', ['203.0.113.183'], '200', '100', '4w2d']
['B', 'IN', '192.0.2.204/30', ['203.0.113.183'], '200', '100', '4w2d']
['B', 'IN', '192.0.2.80/30', ['203.0.113.183'], '200', '100', '4w2d']
['B', 'IN', '192.0.2.208/30', ['203.0.113.183'], '200', '100', '4w2d']This example was somewhat more complex than the average List usage. If there is a clear delimiter between rows, then that can be used for the Record rather than having to do the 'forward match' style here.
By itself, the FSM is not entirely useful. It's intended to be used by Python programs for parsing of textual information. Here we will work through a short example of using the FSM within a Python script.
To start, create a new file called 'routes.py', and add the following into it:
#!/usr/bin/python2.4
import textfsm
import sys
# Open the template file, and initialise a new TextFSM object with it.
template_file = sys.argv[1]
fsm = textfsm.TextFSM(open(template_file))
# Read stdin until EOF, then pass this to the FSM for parsing.
input_data = sys.stdin.read()
fsm_results = fsm.ParseText(input_data)
print 'Header:'
print fsm.header
print 'Prefix | Gateway(s)'
print '-------------------------------'
for row in fsm_results:
print '%-18.18s %s' % (row[2], ', '.join(row[3]))Make it executable, and run it as follows:
$ ./routes.py routes.tmpl < routes.txtExamine the output, and note how each row 'Record'ed in the FSM is represented as a single list type entry in the fsm_results list. Also note that the 3rd entry in the row is a list type, which we connect with a join(). The header, as a list of Value names, is printed also.
Python module which implements a template based state machine for parsing semi-formatted text. Originally developed to allow programmatic access to information returned from the command line interface (CLI) of networking devices.
The engine takes two inputs - a template file, and text input (such as command responses from the CLI of a device) and returns a list of records that contains the data parsed from the text.
A template file is needed for each uniquely structured text input. Some examples are provided with the code and users are encouraged to develop their own.
By developing a pool of template files, scripts can call TextFSM to parse useful information from a variety of sources. It is also possible to use different templates on the same data in order to create different tables (or views).
TextFSM was developed internally at Google and released under the Apache 2.0 licence for the benefit of the wider community.
If you are not a Google employee, our lawyers insist that you sign a Contributor Licence Agreement (CLA).
If you are an individual writing original source code and you're sure you own the intellectual property, then you'll need to sign an individual CLA. Individual CLAs can be signed electronically. If you work for a company that wants to allow you to contribute your work, then you'll need to sign a corporate CLA. The Google CLA is based on Apache's. Note that unlike some projects (notably GNU projects), we do not require a transfer of copyright. You still own the patch.
Sadly, even the smallest patch needs a CLA.
The repo has publicly enabled wiki by Shashank https://twitter.com/cyberboyIndia
TextFSM is a Python module that implements a template based state machine for parsing semi-formatted text. Originally developed to allow programmatic access to information given by the output of CLI driven devices, such as network routers and switches, it can however be used for any such textual output.
The engine takes two inputs - a template file, and text input (such as command responses from the CLI of a device) and returns a list of records that contains the data parsed from the text.
A template file is needed for each uniquely structured text input. Some examples are provided with the code and users are encouraged to develop their own.
By developing a pool of template files, scripts can call textFSM to parse useful information from a variety of sources. It is also possible to use different templates on the same data, in order to create different tables (or views).
The quick-start example:
# Run text through the FSM.
# The argument 'template' is a file handle and 'raw_text_data' is a string.
re_table = textfsm.TextFSM(template)
data = re_table.ParseText(raw_text_data)
# Display result as CSV
# First the column headers
print( ', '.join(re_table.header) )
# Each row of the table.
for row in data:
print( ', '.join(row) )The library is executable directly and can be used to check the syntax of a template and compare input with expected output.
parser.py [--help] template [input_file [output_file]]If installed as a package it will be under the pertinent python version.
/usr/local/lib/pythonx.y/dist-packages/textfsm/parser.pyAt its core, the FSM's purpose is to extract the essential data from a textual data and place it into a tabular representation.
Given the following raw input, we may be interested in line card state and temperature:
Routing Engine status:
Slot 0:
Current state Master
Election priority Master (default)
Temperature 39 degrees C / 102 degrees F
CPU temperature 55 degrees C / 131 degrees F
DRAM 2048 MB
Memory utilization 76 percent
CPU utilization:
User 95 percent
Background 0 percent
Kernel 4 percent
Interrupt 1 percent
Idle 0 percent
Model RE-4.0
Serial ID xxxxxxxxxxxx
Start time 2008-04-10 20:32:25 PDT
Uptime 180 days, 22 hours, 45 minutes, 20 seconds
Load averages: 1 minute 5 minute 15 minute
0.96 1.03 1.03
Routing Engine status:
Slot 1:
Current state Backup
Election priority Backup
Temperature 30 degrees C / 86 degrees F
CPU temperature 31 degrees C / 87 degrees F
DRAM 2048 MB
Memory utilization 14 percent
CPU utilization:
User 0 percent
Background 0 percent
Kernel 0 percent
Interrupt 1 percent
Idle 99 percent
Model RE-4.0
Serial ID xxxxxxxxxxxx
Start time 2008-01-22 07:32:10 PST
Uptime 260 days, 10 hours, 45 minutes, 39 secondsApplying this textual input to the FSM with an appropriate template, the result is returned in a tabular representation where the line card and the relevant data are associated together as fields of a record.
| Slot | Model | Dram | State | Temp | CPUTemp |
|---|---|---|---|---|---|
| 0 | RE-4.0 | 2048 | Master | 39 | 55 |
| 1 | RE-4.0 | 2048 | Backup | 30 | 31 |
The template file is the description of how the FSM should parse out data. Individual templates are required for different patterns of textual input. For example parsing output from router commands generally requires one template for each command.
A full example template follows. In this case, to process the above 'show chassis routing-engine' command output from a router:
# Chassis value will be null for single chassis routers.
Value Filldown Chassis (.cc.?-re.)
Value Required Slot (\d+)
Value State (\w+)
Value Temp (\d+)
Value CPUTemp (\d+)
Value DRAM (\d+)
Value Model (\S+)
# Allway starts in 'Start' state.
Start
^${Chassis}
# Record current values and change state.
# No record will be output on first pass as 'Slot' is 'Required' but empty.
^Routing Engine status: -> Record RESlot
# A state transition was not strictly necessary but helpful for the example.
RESlot
^\s+Slot\s+${Slot}
^\s+Current state\s+${State}
^\s+Temperature\s+${Temp} degrees
^\s+CPU temperature\s+${CPUTemp} degrees
^\s+DRAM\s+${DRAM} MB
# Transition back to Start state.
^\s+Model\s+${Model} -> Start
# An implicit EOF state outputs the last record.The template file consists of two top level sections.
A line is considered a comment if it starts with any optional white space then a hash i.e matches regular expression: "^\s*#".
One or more 'Value' lines are used to describe each column that will be in the resulting table. These Value lines must all appear before any state definitions and must be contiguous lines, separated only by comments.
Each Value line is of the following format:
Value [option[,option...]] name regex
| Keyword | Type | Description |
|---|---|---|
| Value | Keyword | Indicates that this is a Value line entry, mandatory. |
| option | Flags, comma separated (no spaces) | Extra options regarding the value. May be one or more of: Filldown The previously matched value is retained for subsequent records (unless explicitly cleared or matched again). In other words, the most recently matched value is copied to newer rows unless matched again. Key Declares that the fields contents contribute to the unique identifier for a row. Required The record (row) is only saved into the table if this value is matched. List The value is a list, appended to on each match. Normally a match will overwrite any previous value in that row. Fillup Like Filldown, but populates upwards until it finds a non-empty entry. Not compatible with Required. |
| name | Value name | The name of the Value, which will end up as the column name. Must not be the name of a valid option. |
| regex | A regex | The regex against which the Value will match. This regex must be contained within parenthesis. |
After the Value definitions, the State definitions are described. Each state definition is separated by a blank line. The first line is the state name, an alphanumeric word followed by a series of rules.
A state definition is of the following format:
stateName
^rule
^rule
...Multiple state definitions are to be separated by at least one blank line. Rules are described on consecutive lines immediately following the state name and must be indented by one or two white spaces and a carat ('^').
Initially, the FSM will begin at the Start state. Input is only compared to the current state but a matched line can trigger a transition to a new state. Evaluation continues line by line until either EOF is encountered or the current state transitions to the End state.
The FSM starts in state Start, so this label is mandatory and the template will not parse without it.
If EOF was reached on the input then the EOF state is executed. This is an implicit state that looks like:
EOF
^.* -> RecordEOF records the current row before returning. To override this behavior - explicitly define an empty EOF state. like so:
EOFThe End state is reserved and terminates processing of input lines and does not execute the EOF state.
Each state definition consists of a list of one or more rules. The FSM reads a line from the input buffer and tests it against each rule, in turn, starting from the top of the current state. If a rule matches the line, then the action is carried out and the process repeats (from the top of the state again) with the next line.
Rules are of the following format:
^regex [-> action]
regex is a regular expression compared against input lines. The match is performed from the start of the input line, so the carat ('^') although implicit, is required syntax as a reminder of this behavior.
The regex may contain zero or more Value descriptors. Value descriptors are in the format $ValueName or ${ValueName} (the latter format is preferred) and indicate value assignment. The regex of the associated value is substituted into the rule regex, and if the line matches, the text that matches this Value is assigned to the current row. To indicate the end of line (EOL) use a double dollar sign '$$', this will be substituted for a single dollar sign during Value substitution.
For example, take the following template:
Value Interface (\S+)
Start
^Interface ${Interface} is upWhen initially parsing the template, the FSM will expand the rule's regex to:
^Interface (\S+) is upIf the following line is parsed through this rule, then the value 'Interface' is given the value GigabitEthernet1/10:
Interface GigabitEthernet1/10 is up.Multiple value substitutions may be placed into a rule regex. The entire expanded regex must match, for any Values to be assigned.
Following a regexp, actions may be described, delimited by '->' and are of the format 'A.B C'.
Actions are broken down into three optional parts. A) Line Actions, actions on the input line. B) Record Actions, actions on the values collected so far. C) State transition.
If actions are not described i.e. no '->', then the default implicit action is Next.NoRecord.
| Action | Description |
|---|---|
| Next | Finish with the input line, read in the next and start matching again from the start of the state. This is the default behavior if no line action is specified. |
| Continue | Retain the current line and do not resume matching from the first rule of the state. Continue processing rules as if a match did not occur (value assignments still occur). |
After the line action is the optional record action, these are separated by a full stop '.'.
| Action | Description |
|---|---|
| NoRecord | Do nothing. This is the default behavior if no record action is specified. |
| Record | Record the values collected so far as a row in the return data. Non Filldown values are cleared. Note: No record will be output if there are any 'Required' values that are unassigned. |
| Clear | Clear non Filldown values. |
| Clearall | Clear all values. |
The dot '.' separator is only required if both line and record actions are specified. If one or both are left as the implicit default then the dot is omitted i.e. Next, Next.NoRecord and NoRecord are equivalent.
The action can be optionally followed by white spaces and a new State. The State must be one of the reserved states or a valid state defined in the template. Upon matching, after any actions are performed normally, the next line is read from input and the current state is then changed to the new state and processing continues in this new state.
Note that the Continue action does not accept a state transition. This ensures that state machines are loop free.
There is a special action 'Error'. This action will terminate all processing and will not return the table, discarding all rows collected so far, and raises an exception.
The syntax for this action is:
^regex -> Error [word|"string"]