Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-620] Building Documentation with Sphinx 1.3.5 fails: Could not parse literal_block as "swift" #43237

Closed
ahoppen opened this issue Jan 26, 2016 · 23 comments
Labels
bug A deviation from expected or documented behavior. Also: expected but undesirable behavior. documentation

Comments

@ahoppen
Copy link
Collaborator

ahoppen commented Jan 26, 2016

Previous ID SR-620
Radar None
Original Reporter @ahoppen
Type Bug
Status Closed
Resolution Done

Attachment: Download

Environment

Sphinx 1.3.5

Additional Detail from JIRA
Votes 0
Component/s Project Infrastructure
Labels Bug, Documentation
Assignee None
Priority Medium

md5: f29451f0a4f28cf621fa382441a0a9d3

is duplicated by:

  • SR-2213 [sphinx] Could not lex literal_block as "swift".

Issue Description:

Building the Swift documentation with the latest Sphinx (1.3.5 released Jan 24, 2016) fails with the following error message:

Warning, treated as error:
/path/to/swift/docs/ABI.rst:56: WARNING: Could not parse literal_block as "swift". highlighting skipped.

Because the warning is treated as an error, it is currently not possible to build the documentation using the makefile with Sphinx 1.3.5

This is probably related because issue #1565 was fixed in Sphinx (http://www.sphinx-doc.org/en/stable/changes.html)
For a discussion on this problem see: sphinx-doc/sphinx#2264

@belkadan
Copy link
Contributor

cc @jckarter, @dabrahams

@ahoppen
Copy link
Collaborator Author

ahoppen commented Feb 1, 2016

Since the documentation is build by the build script automatically this prevents even basic actions using the build script like creating the Xcode project using

utils/build-script -x -- --reconfigure

@ahoppen
Copy link
Collaborator Author

ahoppen commented Feb 7, 2016

For anyone coming across this issue: To resolve it for now, install Sphinx version 1.3.4, e.g. using

easy_install Sphinx==1.3.4

@swift-ci
Copy link
Collaborator

Comment by Hitster GTD (JIRA)

@ahoppen,
Alternatively, just remove -W from the `SPHINXBUILDOPTS` variable in `docs/Makefile`, as a temporary workaround.

@belkadan
Copy link
Contributor

I finally took a look at this and I'm rather confused, because the literal block does look like valid pygments-Swift. (That is, when I pass it directly to the SwiftLexer defined in utils/pygments/, I get colorized output.)

#!/usr/bin/python
# With PYTHONPATH pointing at swift/utils/pygments/
import swift as swift_pygments_lexers
from pygments import highlight
from pygments.formatters import TerminalFormatter
print(highlight(sys.stdin.read(), swift_pygments_lexers.SwiftLexer(), TerminalFormatter()))

I'll remove the warning-as-error for now.

@belkadan
Copy link
Contributor

Actually, it looks like every highlight is broken. I think someone else really has to look at this properly. I was using this to test what was actually being passed to highlight, and it all looked reasonable.

diff --git a/docs/conf.py b/docs/conf.py
index 9aef6f3..65d5581 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -282,7 +282,7 @@ sys.path.pop(0)
 # "I100 Import statements are in the wrong order." when linting using
 # flake8-import-order. "noqa" is used to suppress this warning.
 from pygments.lexers import get_lexer_by_name as original_get_lexer_by_name  # noqa (E402)
-
+from pygments import highlight as original_highlight  # noqa (E402)
 
 def swift_get_lexer_by_name(_alias, *args, **kw):
     if _alias == 'swift':
@@ -292,5 +292,13 @@ def swift_get_lexer_by_name(_alias, *args, **kw):
     else:
         return original_get_lexer_by_name(_alias, *args, **kw)
 
+def highlight(code, lexer, formatter):
+  print(repr(lexer), repr(formatter))
+  result = original_highlight(code, lexer, formatter)
+  return result
+
+import pygments
+pygments.highlight = highlight
+
 import pygments.lexers  # noqa (I100 Import statements are in the wrong order.)
 pygments.lexers.get_lexer_by_name = swift_get_lexer_by_name

@ahoppen
Copy link
Collaborator Author

ahoppen commented Jun 22, 2016

Thanks for investigating and removing the warnings-as-errors flag. I guess we can remove the note saying that newer versions of sphinx are not supported from README.md in the main repository now, can’t we?

@belkadan
Copy link
Contributor

I didn't actually end up removing it (because of the spew of output). Since the contents of the literal block seem to actually be correct, I'm not sure what's going on.

@belkadan
Copy link
Contributor

Pinging @dabrahams again, since IIRC he was the one to set it up in the first place.

@ahoppen
Copy link
Collaborator Author

ahoppen commented Jun 22, 2016

OK, thanks!

@dabrahams
Copy link
Collaborator

I’m afraid that if I set up syntax highlighting for our docs, I don’t remember anything about it.

@belkadan
Copy link
Contributor

I don't think anyone else has ever worked with Sphinx or Pygment lexers, though. I don't want to just switch to the default Swift lexer (if they have one) because ours is a lot more lax, which makes sense when proposing new things.

Maybe we can just disable this one warning, even if it means syntax highlighting is inconsistent? Or we can drop the syntax highlighting altogether.

@ahoppen
Copy link
Collaborator Author

ahoppen commented Jun 22, 2016

Weren’t there plans to switch to markdown for the documentation anyway? I don’t know where and when I heard it, so I may be wrong.

@swift-ci
Copy link
Collaborator

Comment by Farhan Ahmed (JIRA)

This is also happening with Sphinx 1.4.4. I get the following when building the docs:

Warning, treated as error:
/home/[redacted]/swift/docs/ABI.rst:56: WARNING: Could not lex literal_block as "swift". Highlighting skipped.

Any possible work around to at least let build finish?

@swift-ci
Copy link
Collaborator

Comment by Link Dupont (JIRA)

FWIW, I could reproduce this by calling get_tokens(code) inside the monkeypatched highlight method. But importing SwiftLexer directly via the python REPL, it formats correctly.

@swift-ci
Copy link
Collaborator

Comment by Jason Zavaglia (JIRA)

I've taken a look at this - I think the utils/pygments/swift.py lexer needs a bit of TLC.

Pygments is throwing an error - the grammar specified in the lexer is not fully correct - and the build error is due to Sphinx being more strict post 1.3.4

The very first code block to be attempted to be formatted is this one, which I spent a bit of time in the debugger looking at.

// LLVM <{ i64, i8 }>
struct S {
  var x: Int
  var y: UInt8
}

// LLVM <{ i8, [7 x i8], <{ i64, i8 }>, i8 }>
struct S2 {
  var x: UInt8
  var s: S
  var y: UInt8
}

// LLVM <{}>
struct Empty {}

// LLVM <{ i64, i64 }>
struct ContainsEmpty {
  var x: Int
  var y: Empty
  var z: Int
}

This immediately raises exceptions when trying to parse "var x: Int"

The definition of _isa in utils/pygments/swift.py needs to be changed in the following way - _isa appears to be a pattern for matching <token> : <token>

     _isa = r'([a-zA-Z_][a-zA-Z0-9_]*)(\s+)(:)(\s+)([A-Z0-9_][a-zA-Z0-9_]*)'

to...

     _isa = r'([a-zA-Z_][a-zA-Z0-9_]*)(\s*)(:)(\s*)([a-zA-Z0-9_][a-zA-Z0-9_]*)'

This isn't actually sufficient to get it to work completely, as the lexer is still not terminating correctly at the very end of successfully processing the buffer - not exactly sure why.

Even if I one were to fix this, it would only fix the very first case, and there are likely to be more problems in the grammar which are uncovered with other code snippets in the documentation - so there is probably a fair bit of work and testing to be done here - it would be worth putting some test cases in there I think. I hope that helps gives some clarity on the problem, I may try to come back to it later.

@swift-ci
Copy link
Collaborator

Comment by Jeremy Fergason (JIRA)

I took a look at the lexer and got most of the warnings resolved. This implements a couple of extra lexers for the SIL and ABI code blocks. (see attached patch) lexer.diff

There are still some lingering issues though, mostly with SIL.rst:

WARNING: Definition list ends without a blank line; unexpected unindent.
ERROR: Unexpected indentation.

These seem to revolve around sphinx getting confused about parsing the file.

@belkadan
Copy link
Contributor

Thanks for the patch, Jeremy! (And thanks for the initial triage, Link and Jason.) I finally got back to this and think I've resolved all the issues. See #4283

@ahoppen
Copy link
Collaborator Author

ahoppen commented Aug 13, 2016

If this resolves all the issues (I haven't tested yet), we should change

easy_install -U Sphinx==1.3.4

in README.md to

easy_install -U Sphinx

@belkadan
Copy link
Contributor

Committed to master in ca14c36.

@belkadan
Copy link
Contributor

Waiting for someone else to verify the fix before I update the README.

@ahoppen
Copy link
Collaborator Author

ahoppen commented Aug 16, 2016

I have tested everything with Sphinx 1.4.5. Thanks for fixing this!

@belkadan
Copy link
Contributor

Great! README updated in d318c95.

@swift-ci swift-ci transferred this issue from apple/swift-issues Apr 25, 2022
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug A deviation from expected or documented behavior. Also: expected but undesirable behavior. documentation
Projects
None yet
Development

No branches or pull requests

4 participants