Subject: [Tools-discuss] Table of contents hack

From: Martin Thomson <[email-masked]>
Date: Tue, 10 Jun 2014 12:19:06 -0700

Turns out that xml2rfc formats the TOC as a flat list, which is
inconvenient if you want to do a little reformatting.  Julian's xslt
doesn't suffer this problem, but it's hard to get java everywhere.

The fix is trivial.  And attached.
Index: html.py
===================================================================
--- html.py	(revision 1479)
+++ html.py	(working copy)
@@ -27,7 +27,7 @@
         Uses HTML templates located in the templates directory.
     """
     # HTML Specific Defaults that are not provided in templates or XML
-    html_defaults = { 
+    html_defaults = {
         'references_url': 'http://tools.ietf.org/html/',
     }
 
@@ -37,7 +37,7 @@
         self.iref_index = []
         self.nbws_cond = ' '    # Use space as they are in an anchor element anyway
         self.cref_counter = 0
-        
+
         # Initialize templates directory
         self.templates_dir = templates_dir or \
                              os.path.join(os.path.dirname(xml2rfc.__file__),
@@ -56,13 +56,13 @@
                         'header_rows': [],
                         'toc_head_links': [],
                         'toc_rows': []}
-        
+
         # Use an active buffer pointer
         self.buf = self.buffers['front']
-        
+
         # String holding the final document to write
         self.output = ''
-        
+
         # Temporary div for recursive functions
         self.temp_div = E.DIV()
 
@@ -120,7 +120,7 @@
             format_str = style.partition('format ')[2]
             allowed_formats = ('%c', '%C', '%d', '%i', '%I', '%o', '%x', '%X')
             if not any([ f in format_str for f in allowed_formats]):
-                xml2rfc.log.warn('Invalid format specified: %s ' 
+                xml2rfc.log.warn('Invalid format specified: %s '
                                  '(Must be one of %s)' % (style,
                                     ', '.join(allowed_formats)))
             counter_index = list.attrib.get('counter', None)
@@ -169,8 +169,16 @@
             xml2rfc.log.warn('Invalid tocdepth specified, must be integer:', \
                              tocdepth)
             tocdepth = 3
+        curdepth = 1
         for item in self._getTocIndex():
             if item.level <= tocdepth:
+                # set UL level
+                if item.level > curdepth:
+                    self.buffers['toc_rows'].append('<ul>')
+                elif item.level < curdepth:
+                    self.buffers['toc_rows'].append('</ul>')
+                curdepth = item.level
+
                 # Create link for head
                 link = E.LINK(href='#' + item.autoAnchor)
                 link.attrib['rel'] = 'copyright' in item.autoAnchor and \
@@ -192,7 +200,7 @@
             return lxml.html.tostring(element, pretty_print=True, method='xml', encoding='ascii').decode()
         else:
             return lxml.html.tostring(element, pretty_print=True, method='xml')
-    
+
     def _flush_temp_div(self):
         lines = []
         for child in self.temp_div:
@@ -199,9 +207,9 @@
             lines.append(self._serialize(child))
             child.drop_tree()
         return '\n'.join(lines)
-    
+
     def _expand_ref(self, element):
-        """ Return a list of HTML elements that represent the reference """   
+        """ Return a list of HTML elements that represent the reference """
         if element.tag == 'xref':
             target = element.attrib.get('target', '')
             item = self._getItemByAnchor(target)
@@ -219,7 +227,7 @@
                 return [a, cite]
             else:
                 # Create xref from index lookup
-                format = element.attrib.get('format', 
+                format = element.attrib.get('format',
                                             self.defaults['xref_format'])
                 a = E.A(href='#' + target)
                 if not item:
@@ -347,7 +355,7 @@
             if autoAnchor:
                 a.attrib['href'] = '#' + autoAnchor
             h.append(a)
-        
+
         # Add to body buffer
         self.buf.append(self._serialize(h))
 
@@ -355,7 +363,7 @@
         if text:
             p = E.P(text)
             if autoAnchor:
-                p.attrib['id'] = autoAnchor        
+                p.attrib['id'] = autoAnchor
             # Add to body buffer
             self.buf.append(self._serialize(p))
 
@@ -422,7 +430,7 @@
                 right = right_header[i]
             else:
                 right = ''
-                
+
             # Add row to header_rows buffer
             left_td = E.TD(left)
             left_td.attrib['class'] = 'left'
@@ -434,10 +442,10 @@
     def write_address_card(self, author):
         # Create substitution dictionary with empty string values
         vars = ['fullname', 'surname', 'role', 'organization',
-                'street_spans', 'locality', 'region', 'code', 
+                'street_spans', 'locality', 'region', 'code',
                 'locality_sep', 'region_sep', 'country', 'contact_spans']
         subs = dict(zip(vars, [''] * len(vars)))
-        
+
         # Get values with safe defaults
         subs['fullname'] = author.attrib.get('fullname', '')
         subs['surname'] = author.attrib.get('surname', '')
@@ -446,7 +454,7 @@
         subs['organization'] = author.find('organization') is not None and \
                                author.find('organization').text or ''
         address = author.find('address')
-        
+
         # Crawl through optional elements
         if address is not None:
             postal = address.find('postal')
@@ -502,7 +510,7 @@
                 contact_div.append(span)
 
             # Serialize the temp div
-            subs['contact_spans'] = self._flush_temp_div() 
+            subs['contact_spans'] = self._flush_temp_div()
 
         # Run through template and add to body buffer
         html = self.templates['address_card.html'].substitute(subs)
@@ -536,7 +544,7 @@
                 if surname is not None:
                     if j == len(authors) - 1 and len(authors) > 1:
                         # Last author, render in reverse
-                        name_string = initials + ' ' + surname 
+                        name_string = initials + ' ' + surname
                     else:
                         name_string = surname + ', ' + initials
                     a = E.A(name_string)
@@ -600,7 +608,7 @@
             refkeys = sorted(refkeys)
         for key in refkeys:
             tbody.append(refdict[key])
-        self.ref_start += i + 1                
+        self.ref_start += i + 1
         # Add to body buffer
         self.buf.append(self._serialize(E.TABLE(tbody)))
 
@@ -670,7 +678,7 @@
         # Write the heading
         self.write_heading('Index', autoAnchor='rfc.index')
         table = E.TABLE()
-        # Sort iref items alphabetically, store by first letter 
+        # Sort iref items alphabetically, store by first letter
         alpha_bucket = {}
         keys = list(self._iref_index.keys())
         keys.sort(key=str.upper)
@@ -705,7 +713,7 @@
                     table.append(E.TR(E.TD(' '), td))
 
         self.buf.append(self._serialize(table))
-    
+
     def pre_indexing(self):
         pass
 
@@ -723,9 +731,9 @@
     def post_rendering(self):
         # Create table of contents buffers
         self._create_toc()
-        
+
         # Grab values that haven't been inserted yet
-        background_image = (self.pis['background'] and 
+        background_image = (self.pis['background'] and
             "background-image:url('%s');" % self.pis['background'] or '')
         title = self.r.find('front/title').text
         docName = self.r.attrib.get('docName', '')
@@ -755,8 +763,8 @@
         authors = self._format_author_string(self.r.findall('front/author'))
 
         # Run through base template, store in main output string
-        subs = { 
-                 # Replace flat values 
+        subs = {
+                 # Replace flat values
                  'background':      background_image,
 
                  # HTML-escaped values
@@ -767,7 +775,7 @@
                  'authors':         cgi.escape(authors, quote=True),
                  'keywords':        cgi.escape(', '.join(keyword_list), quote=True),
                  'title':           cgi.escape(title),
-                 
+
                  # Replace buffers
                  'front':           ''.join(self.buffers['front']),
                  'body':            ''.join(self.buffers['body']),

---

From: Julian Reschke <[email-masked]>
Date: Tue, 10 Jun 2014 21:24:16 +0200

On 2014-06-10 21:19, Martin Thomson wrote:
> ...

You don't need Java for it. Just use xsltproc (or msxsl on Windows).

Best regards, Julian

---

From: Martin Thomson <[email-masked]>
Date: Tue, 10 Jun 2014 12:35:37 -0700

> The fix is trivial.  And attached.

The fix is also wrong, at least outside of the narrow set of tests I
initially ran.

Is there any reason this tool can't be put on github, where I could
have submitted a pull request?

> You don't need Java for it. Just use xsltproc (or msxsl on Windows).

Sorry, the complaint should have been leveled at XLST in general, not
the specific tool.
Index: html.py
 
@@ -37,7 +37,7 @@
         self.iref_index = []
 
@@ -120,7 +120,7 @@
+                for i in range(item.level, curdepth):
@@ -187,12 +195,15 @@
                 li = E.LI(counter_text)
                 li.append(a)
                 self.buffers['toc_rows'].append(self._serialize(li))
+        for i in range(1, curdepth):
     def _serialize(self, element):
         if sys.version > '3':
@@ -199,9 +210,9 @@
@@ -219,7 +230,7 @@
@@ -347,7 +358,7 @@
 
@@ -355,7 +366,7 @@
         if text:
             p = E.P(text)
             if autoAnchor:
 
@@ -422,7 +433,7 @@
                 right = right_header[i]
             else:
                 right = ''
-                
+
@@ -434,10 +445,10 @@
@@ -446,7 +457,7 @@
@@ -502,7 +513,7 @@
                 contact_div.append(span)
 
             # Serialize the temp div
 
@@ -536,7 +547,7 @@
@@ -600,7 +611,7 @@
 
@@ -670,7 +681,7 @@
         # Write the heading
@@ -705,7 +716,7 @@

@@ -723,9 +734,9 @@
@@ -755,8 +766,8 @@

                  # HTML-escaped values
@@ -767,7 +778,7 @@

---

From: Julian Reschke <[email-masked]>
Date: Tue, 10 Jun 2014 21:56:29 +0200

> have submitted a pull request?

It's in SVN. Just submit a diff :-)

> Sorry, the complaint should have been leveled at XLST in general, not

Anything specific? There are lots of XSLT 1.0 processors, and they are 
easy to install (if they aren't already there).

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Tue, 10 Jun 2014 15:42:15 -0500

> It's in SVN. Just submit a diff :-)

It should be git :)

> Anything specific? There are lots of XSLT 1.0 processors, and they are easy
> to install (if they aren't already there).

XSLT is a major redeeming factor for XML, and the choice to use XML.
Just sayin'.

Nico
--

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 10 Jun 2014 23:29:22 +0200

Hi Martin,

> The fix is also wrong, at least outside of the narrow set of tests I

Ok.  I'm happy to take a patch when you're happy with it.  And given
what I'm about to say below, I really mean that.  I'm happy when people
provide patches!

> have submitted a pull request?

Yes.  The repository it currently lives on works perfectly fine, and I
don't have the cycles or the inclination to spend time moving it and
the issue tracker it's coupled with.  Projects I maintain on github
stays there, projects I maintain elsewhere stays elsewhere, as long as
things don't break.

I'm perfectly happy taking patches by email, or I can give you access
so that you can branch and commit in the current repository, in much
the same way that we work with the datatracker repository.  (See the
pages on the datatracker wiki: http://trac.tools.ietf.org/tools/ietfdb/wiki
about more than you probably wish to know about that).

However, FWIW, I wouldn't act directly on a pull request that contained
the number of lines with purely whitespace changes that I see in your
attached patch.  I read patches (or pull requests) before I apply them,
and almost all of the lines in the patch are non-significant whitespace
changes.  That means that I have to spend time working through many diff
lines that actually don't contain any code changes, which wastes time
which I already have too little of.  Would it be possible for you to ask
your editor to respect whitespace in edited files, instead of always
stripping end-of-line whitespace, and thus introducing noise in the diff?

	Henrik