[Nasm-bugs] [Bug 3392687] New: nasm allocates 1110 gigabytes of memory (and fails)

[noreply-nasm at dev.nasm.us]

https://bugzilla.nasm.us/show_bug.cgi?id=3392687

            Bug ID: 3392687
           Summary: nasm allocates 1110 gigabytes of memory (and fails)
           Product: NASM
           Version: 2.15 (development)
          Hardware: All
                OS: Other Unix
            Status: OPEN
          Severity: normal
          Priority: Medium
         Component: Assembler
          Assignee: nobody at nasm.us
          Reporter: r+nasm at jasper.la
                CC: chang.seok.bae at intel.com, gorcunov at gmail.com,
                    hpa at zytor.com, nasm-bugs at nasm.us
     Obtained from: Build from source archive using configure

Created attachment 411784
  --> https://bugzilla.nasm.us/attachment.cgi?id=411784&action=edit
Initialize strlist_entry

With nasm 2.15.01 on OpenBSD 6.7 with clang 8.0.1, it fails to compile some
larger files with an error stating it cannot allocate memory.
Looking at the ktrace it ends up failing on mmap(2) calls such as these:

CALL 
mmap(0,0x1158a03e000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)

That seems like a rather sizable allocation it attempts to make which ends up
failing. Whilst attempting to debug this I rebuild nasm with '-g -O0' and the
errors were gone.

When built with the default '-O2' I traced this to strlist_vprintf() which is
when I noticed this warning emitted by clang:

nasmlib/strlist.c:111:31: warning: variable 'e' is uninitialized when used here
[-Wuninitialized]
        e = nasm_vaxprintf(offsetin(*e, str), fmt, ap);
                                     ^
./include/compiler.h:196:42: note: expanded from macro 'offsetin'
# define offsetin(p,m)  ((const char *)&((p).m) - (const char *)&(p))
                                          ^
nasmlib/strlist.c:105:25: note: initialize the variable 'e' to silence this
warning
        struct strlist_entry *e;
                               ^
                                = NULL

After adjusting the code to "struct strlist_entry *e = NULL" these bogus
allocations were resolved and nasm runs fine again.

I've attached the patch to resolve this issue.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are watching all bug changes.