ITWriting

Working with large XML files

Administrator — Sun, 17 Jul 2011 05:34:29 +0000

The current task at PMC-Sierra involves finding a better way of publishing register data information, as well as writing scripts to integrate the register information with the rest of the documentation set.

The integration part isn’t a particularly large challenge; I’ve already written scripts the migrate the docset’s Docbook information into DITA to take advantage of both the DITA Open Toolkit’s Eclipse Help publishing driver and Scott Prentice’s AIR Help driver. Since I also want to publish the register data as Docbook, I’ve added scripts that transform the register data into Docbook, and from there I flip it into DITA. The only real challenges are:

Figuring out the most helpful way of presenting the register data (more on that in another entry).
Dealing with the sheer size of the register data files.

Problem #2 manifests itself in all manner of annoying ways. I’m used to using an XML-aware editor, but forget that; there’s not an editor on the market that deals gracefully – if at all – with truly large files (I’m not going to go into size details because…it’s a detail; the files are big. BIG). I can take samples and run scripts against them, but if I want to see how my script will run against reality it’s a 10 hour process that I put in motion before I leave work.

I’m shopping around for a faster test machine with more processor and RAM, but overall I see this as simply a test to my attention to detail. I’ve grown used to being able to rapidly test and prototype and this is like being back in the land of punchcards where I have to hand my cards off to a tech and find out in the morning that I misplaced a colon.

Cygwin Linux Environment for Windows

Administrator — Sun, 17 Jul 2011 05:18:47 +0000

Regardless of which side of the Microsoft/Linux divide you happen to fall on (if you fall on either side at all; I prefer neutrality, myself), it’s tough to dispute that Linux offers a far more developed command line environment than Windows.

Therefore, a tool like Cygwin, which creates a Linux-like command line environment on any Windows machine is a nice tool indeed. Installing Cygwin (http://www.cygwin.com/) and then adding it to your machine’s path variable makes a convincing set of Linux commands available within your Windows command shell.

This is useful both for simple convenience when working in a mixed environment and the creation of cross-platform scripts.

And it’s free, and I like that.

DeltaXML, changebars, and Docbook

Administrator — Sun, 17 Jul 2011 05:17:23 +0000

One of the most important things to understand as a Technical Communications manager when you’re transitioning your company documentation and documentation process away from being Microsoft Word based and towards a centralized CMS is that you’re also transitioning away from being a departmental speedbump and towards being a potential barricade. This transition can be terrifying at certain moments, like the moments when a critical document needs to be published and your CMS is spewing error messages rather than neatly formatted PDFs. This happened to me while ago.

When attempting to publish a differenced version of a docbook publication which had been created with the most excellent DeltaXML Core tool, the XEP processor began spewing “com.renderx.xep.lib.InternalException: No matching point found for change-bar” along with details about exactly which change-bar was lacking a companion change-bar-end. Digging into the FO code revealed that there was indeed a matching change bar end for problematic opening tag, and we were off to the races, ’cause a missing closing tag could be a problem in my XSLT, but when the end is there an XEP can’t find it, it looks more like a bug in XEP, which is roughly a thousand times more frightening.

I can patch my XSLT. I can’t patch XEP.

Two days worth of troubleshooting later it turned out that the problem was a known bug within XEP interacting with some extra spaces added to the output XSL:FO by the DeltaXML Docbook XSLT. It wouldn’t happen everywhere in the document because it only crops up in conjuction with certain kinds of rollbacks that XEP performs to satisfy keep restraints on orphens, etc. The solution was pretty simple. I had to modify the DeltaXML XSLT to add a zero-width character () immediately before and after every change-bar-end.

The document published, the screaming stopped, my ears healed, we all moved on.

Pixware XMLmind XML Editor

Administrator — Sat, 16 Jul 2011 06:28:03 +0000

There are ways in which I might be construed as one of the best friends that the XMLmind XML Editor has ever had, since I’ve been pivotal in arranging for its adoption by not one but two CMS providers (Siberlogic and Bluestream). That said, as much as I think it’s a good editor for creating XML content, my decisions have always been about the Benjamins.

Arranging for editors is always a sticking point when rolling out a CMS, since the editor that you’re usually replacing (Microsoft Word) is already paid for, already on the desktop, and usually included as an inseparable part of the support agreement with Microsoft. Therefore, the prospect of paying another $400-$1000 per seat for every single content contributor and ongoing support cost for yet another editing tool gets the bean

counters gagging.

CMS suppliers, however, can buy a developer license for XMLmind for less than $15,000USD. This license includes unlimited redistribution rights to their customers. In both of the cases that I’ve managed, my customer has fronted the upfront cost of the license and has therefore gained access to global XMLmind licenses via their relationship with the CMS vendor. In the case of PMC-Sierra, this amounts to around 1000 potential users. Fifteen dollars per user. Xmetal is more powerful, and OxygenXML is emerging as the best of the next generation, but just try matching that price point

XMLStarlet

Administrator — Sat, 16 Jul 2011 06:26:30 +0000

As their Sourceforge site states, “XMLStarlet is a set of command line utilities (tools) which can be used to transform, query, validate, and edit XML documents and files using simple set of shell commands in similar way it is done for plain text files using UNIX grep, sed, awk, diff, patch, join, etc commands”. Why does this matter? Because my current project involves working on XML files that are simply too large to open in any of my validating editors. I’m mostly using it to make xpath queries and then validate my output against a RelaxNG schema. It’s free, effective, and fast. Very cool.

OxygenXML 10

Administrator — Sat, 16 Jul 2011 06:25:55 +0000

There are mountains of XML editors available these days, and most of them stink, or are at least indistinguishable from each other. The Oxygen XML Editor does not stink and easily distinguishs itself from the pack by releasing constant upgrades and improvements that often seem to be ripped from my personal list of complaints about XML editors in general.

I primarily use Oxygen as an XSL development environment, and other than the problems with large files that all schema-aware XML editors seem to suffer from, it’s a joy. It’s easy to parameterize to fit specific environmental restrictions, its behaviour is consistent with the restrictions I define (not always the case in other IDEs I’ve used) and its easy to update with new components, schemas, and toolkits.

I don’t often use the Author component of Oxygen, because I my site uses XMLmind and I like to eat the same dogfood as my user community, but whenever I’ve fired it up it shows me features that I wish XMLmind included (particularly the change tracking implementation). In short, when XML is the subject, Oxygen is my favourite object.

Bluestream XDocs 2.1

Administrator — Sat, 16 Jul 2011 06:25:21 +0000

There’s a lot I could write about Bluestream and the XDocs product. The 1.1 version was…not well suited to the setting that we tried to deploy it into. The 2.1 version, however, is nearly as fast, stable, and pleasing as the 1.1 version was slow, buggy, and frustrating.

It requires users to learn new ways of doing things. I don’t apologize for that; change requires change, and this change is progress. It supports DITA beautifully and is adaptable to other schemas with some fairly minor development. The bundled XMLmind editor combined with the low licensing cost makes this system the undisputed value leader in its space. Any sysadmin who takes the time needed to learn how to leverage the combination of the two tools (XDocs and XMLmind) can’t help but be impressed with what this system can do for such a stunningly low price.

I don’t work for Bluestream, have received no considerations from Bluestream, and have, in fact, directed a large volume of money towards Bluestream. If I have my way, I’ll be directing more.

Rack’em Stack’em and Pack’em

Administrator — Sat, 16 Jul 2011 06:23:37 +0000

One of the issues when working with large datasets is that it can sometimes result in uncomfortably large output documents. If some of the documents that my users work with were produced as PDFs they’d be well over 10,000 pages long.

Recently, I was asked if I could take a generated table of register addresses and rework the XSLT so that it was broken off into a series of tables of no more than 500 entries each. I’d never done anything like this before, so it took me a few trys to get it right, but here it is. The following script will grab every node of element type “testnode”, read it into variable x, and then parse it out in 500-piece chunks. This may well be a perfectly ordinary thing to do, but I couldn’t find any good examples of it before I started writing this one, so I thought I’d post it on the odd chance that someone else found it useful.

Hyphenation in Docbook PDF Tables

Administrator — Sat, 16 Jul 2011 06:23:02 +0000

Hyphenation is complicated. Better minds than mine have detailed why in a great swack of places, so I’m not going to repeat the reasons here. The reason I mention this at all is that one of the places where the complicated nature of hyphenation manifests is within tables (because they sport short line lengths and therefore force more hyphenation decisions per word) and I’ve recently came across a more elegant solution than I was previously using, which was simply to add zero-width spaces after every single character in every single table entry.

At PMC-Sierra, hyphenation within table cells is particularly problematic because many of the table cell entries are in formats that resist hyphenation: URLs, field names, formulas, and other types of content that have no clues that would enable a normal hyphenation routine to figure out where the break should be. The result of this is that in the unmodified Docbook XSL publishing scripts, these entries resist hyphenation and simply bleed into the next cell. Not a popular option with the reader community.

The template below can be inserted into your modification layer and addresses the vast majority of hyphenation corner cases. It does this by reading the contents of every table cell and inserting a potential hyphenation break after every type of character found in the tableentry.hyphenate.chars parameter.

In the event that WordPress messes up the following XSLT enough to make is unusable, just take a look at the hyphenate-URL template in the stock DocbookXSL (I’m using 1.7.3) style sheets. It’s just a minor tweaking to apply it to table entries.

I think Bob Stayton pointed this one out to me. Use and enjoy.
























_-;:

Visio, SVG, and making it work with RenderX XEP

Administrator — Sat, 16 Jul 2011 06:22:13 +0000

The SVG (Scalable Vector Graphic) standard is one of those infuriating technologies that appears perpetually poised on the brink of real usefulness. In theory, it’s wonderful; an XML schema for describing shapes, which means a way of generating visual data off of XML data.

In context to what I’m doing at PMC-Sierra, the potential uses are almost limitless. PMC’s products, like most others in the space, double in density every 4 years or so. The current product has approx 400 million transistors. Do the math. The document describing the chip’s basic operating system is over 8000 pages in PDF form. I ran across some engineers the other day making changes that would have blown out the document to over 40,000 pages. Is that even possible? A 40,000 page PDF?

We’re now hitting a tipping point where the sheer scale of the documentation set is a serious barrier to its use. We need to increase the density of the information itself in order to prevent drowning our users in reams information that is irrelevant to the tiny little corner of the product that they are actually concerned with. What’s the best way of doing this? Visual aids! Let’s generate interrupt path graphics right off the RDL (Register Description Language) for the chip! Let’s link TSB (Telecom Service Block) programming content right to an image of the chip itself, right to the CAD drawing of the chip’s physical layout! And let’s automate these relationships so that we don’t kill some lowly co-op with boredom, manually making and repairing the links! Exciting stuff, and it’s just the tip of the iceberg.

Before we get there, though, let’s start small; let’s export the hundreds of hideously complex diagrams that we’ve created in Visio into SVG, rather than high-resolution JPGs or PNGs, and publish them in our XML content management system. It’ll be great; the documents will be smaller (vector trumps raster!) and will be better quality, since users will be able to zoom, pan, and generally get up close and personal with the images in a way that they can’t with raster formats.

Small problem; Microsoft Visio does not export any kind of SVG that anything other than Microsoft recognizes. It turns out that SVG has a long and complicated family tree, and that while Adobe has IE plug-in that “supports” SVG and Chrome, FireFox, and Opera all “support” SVG, they don’t agree on what SVG is except to say that Visio’s SVG is not proper SVG. Or maybe it is, maybe the problem is actually with Batik, the open source technology used to render SVG, maybe that’s the problem. From an XML print production point of view FOP doen’t appear to support SVG at all, but RenderX XEP does to the same extent that the Adobe plugin does (which falls short of real usefulness). Our CMS uses XEP, I’m back within sight of usefullness, but not yet at the place itself. I’m still stuck with SVG images that are 95% right but 5% wrong and a bunch of software providers pointing fingers at each other.

From an 80/20 point of view, I’d be happy if I could get publishing working with XEP and get the images rendering correctly in IE. Firefox, Opera, and Chrome can wait. From that point the problem is pretty boundable: Visio exports SVG files without including the xlink namespace correctly, and a known Batik bug prevents arrowheads from showing up. Therefore, if we write an export macro for Visio that saves to SVG, then reopens the SVG and makes some pretty simple changes, we should be good to go.

And indeed we are. The following visio macro makes the needed changes and, as far as I’ve tested, creates SVGs that publish flawlessly XEP and display flawlessly in IE. Use and enjoy; if you improve it or create a version that exports SVGs compatible for with the Firefox set, please share!

' Module: SVGExportModule
' A collection of functions to export and process SVG files
'
' Revision History
' Number Date       Name  Description
' 1.00   2009/07/24 TC    Initial release.
'
' 1.10   2010/03/24 TC    Updated to fix font-size issue,
'                         test special characters,
'                         automatically add command bar.
'
' 1.20   2010/03/25 TC    Corrected the macro to load the CommandBar
'                         according to Visio's process:
'                         1) Load macro to ThisDocument
'                         2) Added an on document open macro using the property:
'                            "Document_DocumentOpened(ByVal doc As IVDocument)"
'                         3) Added an on document close macro using the property
'                            "Document_BeforeDocumentClose(ByVal doc As IVDocument)"
'                         Added an optional argument to CommandBarAdd to exit
'                         after removing the toolbar.
'
' ' Contents
' Main function
' - SVGExport: Saves current file or selection as SVG,
'              then opens file as text string in order
'              to process search and replace operations
'
' File Common Dialog Library
' - GetSaveFileName (private function)
' - ShowSaveFileDialog
'
' CommandBar functions
' - See ThisDocument.Document_DocumentOpened(ByVal doc As IVDocument)
' - CommandBarAdd: adds CommandBar
' - CommandBarClear: removes CommandBar
'
' Utility functions
' - GetText: Open a text file to store into a string
' - OutputText: Write a string to a text file
' - ProcessText: search and replace character strings in text
' - SplitName: separates filename from its extension  ''======================================================
'' Library:    File Common Dialog Library
'' Desc:       Compensates for Visio's lack of the
''             common MSO FileDialog
'' Call:       ShowSaveFileDialog()
'' Arguments:  sFilter--File name filter (required)
''             sDefExt--Default extension
''             sInitDir--Initial directory
''             lFlags--File flag combinations. See Common Dialogs Help
''             hParent--Handle of he parent form
'' Comments:   FileName is same as the current file
'' Changes----------------------------------------------
'' Date      Programmer     Change
'' 20010421  Marco Belinaso http://www.devx.com/vb2themax/Tip/19260
''====================================================== Private Declare Function GetSaveFileName Lib "comdlg32.dll" Alias _
"GetSaveFileNameA" (pOpenfilename As OpenFilename) As Long Private Type OpenFilename
lStructSize As Long
hwndOwner As Long
hInstance As Long
lpstrFilter As String
lpstrCustomFilter As String
nMaxCustFilter As Long
iFilterIndex As Long
lpstrFile As String
nMaxFile As Long
lpstrFileTitle As String
nMaxFileTitle As Long
lpstrInitialDir As String
lpstrTitle As String
flags As Long
nFileOffset As Integer
nFileExtension As Integer
lpstrDefExt As String
lCustData As Long
lpfnHook As Long
lpTemplateName As String
End Type Private Enum OFNFlagsEnum
OFN_ALLOWMULTISELECT = &H200
OFN_CREATEPROMPT = &H2000
OFN_ENABLEHOOK = &H20
OFN_ENABLETEMPLATE = &H40
OFN_ENABLETEMPLATEHANDLE = &H80
OFN_EXPLORER = &H80000
OFN_EXTENSIONDIFFERENT = &H400
OFN_FILEMUSTEXIST = &H1000
OFN_HIDEREADONLY = &H4
OFN_LONGNAMES = &H200000
OFN_NOCHANGEDIR = &H8
OFN_NODEREFERENCELINKS = &H100000
OFN_NOLONGNAMES = &H40000
OFN_NONETWORKBUTTON = &H20000
OFN_NOREADONLYRETURN = &H8000
OFN_NOTESTFILECREATE = &H10000
OFN_NOVALIDATE = &H100
OFN_OVERWRITEPROMPT = &H2
OFN_PATHMUSTEXIST = &H800
OFN_READONLY = &H1
OFN_SHAREAWARE = &H4000
OFN_SHAREFALLTHROUGH = 2
OFN_SHARENOWARN = 1
OFN_SHAREWARN = 0
OFN_SHOWHELP = &H10
End Enum ' All functions must appear after Declare and Enum ' Show the common dialog to select a file to save. Returns the path of the
' selected file or a null string if the dialog is canceled ' Parameters: ' Example:
'    Dim sFilter As String
'    'set the filter: show text files and all the files
'    sFilter = "Text files (*.txt)|*.txt|All files (*.*)|*.*"
'    'let the user select a file, an ask for confirmation if the file already
' exists
'    MsgBox "File selected: " & ShowOpenFileDialog(sFilter, "txt",
'  "C:\Documents", OFN_OVERWRITEPROMPT) Function ShowSaveFileDialog(Optional ByVal sFilter As String, Optional ByVal sDefExt As _
String, Optional ByVal sInitDir As String, Optional ByVal lFlags As Long, _
Optional ByVal hParent As Long) As String
Dim OFN As OpenFilename
On Error Resume Next
' set the values for the OpenFileName struct
With OFN
.lStructSize = Len(OFN)
.hwndOwner = hParent
.lpstrFilter = Replace(sFilter, "|", vbNullChar) & vbNullChar
.lpstrFile = Space$(255) & vbNullChar & vbNullChar
.nMaxFile = Len(.lpstrFile)
.flags = lFlags
.lpstrInitialDir = sInitDir
.lpstrDefExt = sDefExt
End With
' show the dialog
If GetSaveFileName(OFN) Then
' extract the selected file (including the path)
ShowSaveFileDialog = Left$(OFN.lpstrFile, InStr(OFN.lpstrFile, _
vbNullChar) - 1)
End If
End Function ''======================================================
'' Program:    CommandBarAdd
'' Desc:       Add CommandBar - MS Office 2003 only
''             Visio doesn't use the Ribbon UI
'' Call:       CommandBarAdd
'' Arguments:  None
'' Comments:   Note: should Visio ever change capability
''             this function will need to change
'' Changes----------------------------------------------
'' Date      Programmer   Change
'' 20090724  Tony Chung   Written
'' 20100326  Tony Chung   Added the bRemove boolean
''                        argument to immediately exit
''                        after clearing toolbar
''======================================================
Sub CommandBarAdd(Optional bRemove As Boolean = False)
Dim cbrCmdBar As CommandBar, sCMdBarName As String
'' Create temporary toolbar that doesn't
'' persist between application sessions.
sCMdBarName = "SVGExport"
Call CommandBarClear(sCMdBarName) '' bRemove is set to True BeforeDocumentClose
'' this method ensures we close the same toolbar opened
'' by this macro (same variable name)
If bRemove = True Then GoTo CleanExit     Set cbrCmdBar = Application.CommandBars.Add(Name:=sCMdBarName, Temporary:=True)
cbrCmdBar.Position = msoBarTop '' Add a button to MyDrawingCommandBar
'' that runs a VBA macro.
Set cbButton = cbrCmdBar.Controls.Add(Type:=msoControlButton)
With cbButton
.Caption = "Export SVG file"
.TooltipText = "Export SVG file that conforms to PMC schema"
.Style = msoButtonIconAndCaption
'' Use the Tag property for context switching and
'' for use with the FindControl method.
.Tag = "cbbVBAMacro"
'' Set the button face to use an internal icon.
.FaceID = 7075 '' Use the OnAction property to run a VBA macro
'' contained in this document.
.OnAction = "SVGExport"
End With
cbrCmdBar.Visible = True
CleanExit:
End Sub ''======================================================
'' Program:    CommandBarClear
'' Desc:       Remove specific CommandBar
'' Call:       CommandBarClear(sName)
'' Arguments:  sName--The CommandBar name to remove
'' Comments:   Note: should Visio ever change capability
''             this function will need to change
'' Changes----------------------------------------------
'' Date      Programmer   Change
'' 20090724  Tony Chung   Written
''======================================================
Private Sub CommandBarClear(sName As String)
''   Before adding a new CommandBar check for its
''   existence and remove itto avoid causing an error.     Dim cmdBars As CommandBars
Set cmdBars = Application.CommandBars     For i = 1 To cmdBars.Count
''    Debug.Print i & ": " & cmdBars(i).NameLocal
If cmdBars(i).NameLocal = sName Then
cmdBars(i).Delete
End If
Next i
End Sub ''======================================================
'' Program:    SVGExport
'' Desc:       Save as SVG, reopen to replace xmlns
''             and save as final SVG
'' Call:       SVGExport()
'' Arguments:  None.
'' Comments:   FileName is same as the current file
'' Changes----------------------------------------------
'' Date      Programmer   Change
'' 20090724  Tony Chung   Written
''====================================================== Sub SVGExport()
Dim sPath As String, sFileName As String, sFile As String
Dim sFName As String, sText As String, sRv As String '' establish page container
ActivePage.ResizeToFitContents
'' name file
sPath = ActiveDocument.Path
sFileName = SplitName(ActiveDocument.Name)
sFile = sPath & "\" & sFileName & ".svg"
'' hook into file dialog picker here
sFName = ShowSaveFileDialog( _
sFilter:="Scalable Vector Graphics (*.svg)", _
sDefExt:="svg", sInitDir:=sPath, _
lFlags:=OFN_OVERWRITEPROMPT)     If Trim(sFName) = "" Then
MsgBox "Operation cancelled."
GoTo GracefulExit
End If     sFile = sFName
'' export SVG
Application.ActiveWindow.Selection.Export sFile '' open file
sText = GetText(sFile)
'' process file
sText = ProcessText(sText)
'' write file
sRv = OutputText(sFile, sText)
If sRv = "Success" Then
MsgBox "File: " & sFile & " exported successfully"
Else
MsgBox "Error: " & sRv
End If
GracefulExit:
End Sub ''======================================================
'' Program:    GetText
'' Desc:       Read a text file into a string and then
''             return the string
'' Called by:  ParseText
'' Call:       GetText(sFile)
'' Arguments:  sFile--The full path to the text file
'' Comments:
'' Changes----------------------------------------------
'' Date      Programmer   Change
'' 6/14/06   Charley Kyd  Written
''======================================================
Function GetText(sFile As String) As String
Dim nSourceFile As Integer, sText As String ''Close any open text files
Close ''Get the number of the next free text file
nSourceFile = FreeFile ''Write the entire file to sText
Open sFile For Input As #nSourceFile
sText = Input$(LOF(1), 1)
Close #nSourceFile    GetText = sText
End Function ''======================================================
'' Program:    OutputText
'' Desc:       Print the contents of a string into
''             a text file
'' Called by:
'' Call:       OutputText(sFile, sText)
'' Arguments:  sFile--The full path to the text file
''             sText--The text string to write
'' Comments:
'' Changes----------------------------------------------
'' Date      Programmer   Change
'' 6/14/06   Charley Kyd  Written as GetText()
'' 7/24/09   Tony Chung   Repurposed to OutputText()
''                        Add basic error handling
''======================================================
Function OutputText(sFile As String, sText As String) As String
Dim nSourceFile As Integer, sRv As String '' Close any open text files
Close '' Get the number of the next free text file
nSourceFile = FreeFile '' Write the entire file to sText
On Error GoTo ErrHandler
Open sFile For Output As #nSourceFile
Print #nSourceFile, sText
Close #nSourceFile
GoTo CleanExit
ErrHandler:
sRv = Error CleanExit:
On Error GoTo 0
If sRv = "" Then sRv = "Success"
OutputText = sRv
End Function ''======================================================
'' Program:    ProcessText
'' Desc:       Add custom attributes to xml tags and
''             return the string
'' Called by:
'' Call:       ProcessText(sText)
'' Arguments:  sText--The xml string
'' Comments:   Note: should Visio ever change capability
''             this function will need to change
'' Changes----------------------------------------------
'' Date      Programmer   Change
'' 20090724  Tony Chung   Written
'' 20100324  Tony Chung   Added handler to standardize
''                        font-size between browsers
''====================================================== Function ProcessText(sText As String) As String
'' sRpl( lookup To replace, lookup value To replace value)
Dim sRpl(1 To 3, 1 To 2) As String
sRpl(1, 1) = "xmlns=""http://www.w3.org/2000/svg"""
sRpl(1, 2) = "xmlns=""http://www.w3.org/2000/svg"" xmlns:xlink=""http://www.w3.org/1999/xlink"""
sRpl(2, 1) = "