Add consvar for TEMPFILE encoding

Alternate approach to avoid decoding problems: don't try to guess how
to encode the TEMPFILE.  Leave it utf-8 unless TEMPFILEENCODING is set.

Add comment to TEMPFILEDIR doc on the topic as well.

Signed-off-by: Mats Wichmann <mats@linux.com>

Author: mwichmann

CHANGES.txt

@@ -83,9 +83,9 @@ RELEASE  VERSION/DATE TO BE FILLED IN LATER
       Windows 11 and GitHub windows-2025.
 
   From William Deegan:
-    - Fix Issue #4746. TEMPFILE's are written with utf-8 encoding, this doesn't work
-      for all situations. Change to use locale.getpreferredencoding(False), which should
-      yield the encoding for the system.
+    - Fix Issue #4746. TEMPFILE's are written with utf-8 encoding, In case
+      of decoding errors, TEMPFILEENCODING can now be specified to give
+      more explicit instructions to SCons.
 
   From Edward Peek:
     - Fix the variant dir component being missing from generated source file

RELEASE.txt

@@ -76,9 +76,13 @@ FIXES
 - Fix a test problem on Windows where UNC tests failed due to incorrect path
   munging if a non-default %TEMP% was defined (as in moving to a Dev Drive).
 
-- Fix Issue #4746. TEMPFILE's are written with utf-8 encoding, this doesn't work
-  for all situations. Change to use locale.getpreferredencoding(False), which should
-  yield the encoding for the system.
+- Fix Issue #4746. The TEMPFILE is written in utf-8 encoding by default.
+  If the tempfile contents cannot be decoded by the command the
+  tempfile is passed to, (new) TEMPPFILEENCODING can be used to speficy a
+  different encoding to use. On Windows, the username may be a cause of this,
+  as the default path for temporary files includes the username. Setting
+  (existing) TEMPFILEDIR may also help in this case.
+
 
 IMPROVEMENTS
 ------------

SCons/Platform/Platform.xml

@@ -395,9 +395,18 @@ Note this value is used literally and not expanded by the subst logic.
 <summary>
 <para>
 The directory to create the long-lines temporary file in.
-If unset, some suitable default should be chosen.
-The default tempfile object lets the &Python;
-<systemitem>tempfile</systemitem> module choose.
+If unset, the &Python;
+<systemitem>tempfile</systemitem> module chooses the directory
+based on the <envar>TMPDIR</envar>,
+<envar>TEMP</envar>
+or <envar>TMP</envar> environment variables.
+If the default path causes processing errors,
+set &cv-TEMPFILEDIR; to a safer alternative.
+For example, on Windows,
+the default temporary file path contains the username.
+If the username contains non-7-bit-ASCII characters,
+there may decoding errors opening the path to the temporary file.
+See also &cv-link-TEMPFILEENCODING;.
 </para>
 </summary>
 </cvar>
@@ -411,8 +420,8 @@ If you need to apply extra operations on a command argument
 (to fix Windows slashes, normalize paths, etc.)
 before writing to the temporary file,
 you can set the &cv-TEMPFILEARGESCFUNC; variable to a custom function.
-Such a function takes a single string argument and returns
-a new string with any modifications applied.
+The function must accept a single string argument and
+and return a new string with any modifications applied.
 Example:
 </para>
 
@@ -435,4 +444,23 @@ env["TEMPFILEARGESCFUNC"] = tempfile_arg_esc_func
 </summary>
 </cvar>
 
+<cvar name="TEMPFILEENCODING">
+<summary>
+<para>
+By default, the long-lines temporary file (aka "response file") created by the
+&cv-link-TEMPFILE; function will be encoded in the &Python;
+default encoding, UTF-8.
+If the external command which reads the response file
+encounters decoding errors
+(usually, if that command depends on Windows legacy code pages,
+and a pathname in the response file
+or the response file path itself cannot
+be represented in the 7-bit ASCII characer set),
+set this variable to the appropriate codec.
+See also &cv-link-TEMPFILEDIR;.
+</para>
+<para><emphasis>New in version NEXT_RELEASE</emphasis></para>
+</summary>
+</cvar>
+
 </sconsdoc>

SCons/Platform/__init__.py

@@ -47,11 +47,13 @@
 import os
 import sys
 import tempfile
-import locale
 
+import SCons.Action
 import SCons.Errors
+import SCons.Platform
 import SCons.Subst
 import SCons.Tool
+import SCons.Util
 
 
 def platform_default():
@@ -257,7 +259,8 @@ def __call__(self, target, source, env, for_signature):
         else:
             tempfile_dir = None
 
-        fd, tmp = tempfile.mkstemp(suffix, dir=tempfile_dir, text=True)
+        # default is binary - encode the tempfile contents later
+        fd, tmp = tempfile.mkstemp(suffix, dir=tempfile_dir)
         native_tmp = SCons.Util.get_native_path(tmp)
 
         # arrange for cleanup on exit:
@@ -280,7 +283,8 @@ def tmpfile_cleanup(file) -> None:
         tempfile_esc_func = env.get('TEMPFILEARGESCFUNC', SCons.Subst.quote_spaces)
         args = [tempfile_esc_func(arg) for arg in cmd[1:]]
         join_char = env.get('TEMPFILEARGJOIN', ' ')
-        os.write(fd, bytearray(join_char.join(args) + "\n", encoding=locale.getpreferredencoding(False)))
+        encoding = env.get('TEMPFILEENCODING', 'utf-8')
+        os.write(fd, bytes(join_char.join(args) + "\n", encoding=encoding))
         os.close(fd)
 
         # XXX Using the SCons.Action.print_actions value directly

doc/generated/variables.gen

@@ -9714,8 +9714,8 @@ If you need to apply extra operations on a command argument
 (to fix Windows slashes, normalize paths, etc.)
 before writing to the temporary file,
 you can set the &cv-TEMPFILEARGESCFUNC; variable to a custom function.
-Such a function takes a single string argument and returns
-a new string with any modifications applied.
+The function must accept a single string argument and
+and return a new string with any modifications applied.
 Example:
 </para>
 
@@ -9758,10 +9758,39 @@ Note this value is used literally and not expanded by the subst logic.
     </term>
     <listitem><para>
 The directory to create the long-lines temporary file in.
-If unset, some suitable default should be chosen.
-The default tempfile object lets the &Python;
-<systemitem>tempfile</systemitem> module choose.
-</para>
+If unset, the &Python;
+<systemitem>tempfile</systemitem> module chooses the directory
+based on the <envar>TMPDIR</envar>,
+<envar>TEMP</envar>
+or <envar>TMP</envar> environment variables.
+If the default path causes processing errors,
+set &cv-TEMPFILEDIR; to a safer alternative.
+For example, on Windows,
+the default temporary file path contains the username.
+If the username contains non-7-bit-ASCII characters,
+there may decoding errors opening the path to the temporary file.
+See also &cv-link-TEMPFILEENCODING;.
+</para>
+</listitem>
+  </varlistentry>
+  <varlistentry id="cv-TEMPFILEENCODING">
+    <term>
+      <envar>TEMPFILEENCODING</envar>
+    </term>
+    <listitem><para>
+By default, the long-lines temporary file (aka "response file") created by the
+&cv-link-TEMPFILE; function will be encoded in the &Python;
+default encoding, UTF-8.
+If the external command which reads the response file
+encounters decoding errors
+(usually, if that command depends on Windows legacy code pages,
+and a pathname in the response file
+or the response file path itself cannot
+be represented in the 7-bit ASCII characer set),
+set this variable to the appropriate codec.
+See also &cv-link-TEMPFILEDIR;.
+</para>
+<para><emphasis>New in version NEXT_RELEASE</emphasis></para>
 </listitem>
   </varlistentry>
   <varlistentry id="cv-TEMPFILEPREFIX">

doc/generated/variables.mod

@@ -591,6 +591,7 @@ THIS IS AN AUTOMATICALLY-GENERATED FILE.  DO NOT EDIT.
 <!ENTITY cv-TEMPFILEARGESCFUNC "<envar xmlns='http://www.scons.org/dbxsd/v1.0'>$TEMPFILEARGESCFUNC</envar>">
 <!ENTITY cv-TEMPFILEARGJOIN "<envar xmlns='http://www.scons.org/dbxsd/v1.0'>$TEMPFILEARGJOIN</envar>">
 <!ENTITY cv-TEMPFILEDIR "<envar xmlns='http://www.scons.org/dbxsd/v1.0'>$TEMPFILEDIR</envar>">
+<!ENTITY cv-TEMPFILEENCODING "<envar xmlns='http://www.scons.org/dbxsd/v1.0'>$TEMPFILEENCODING</envar>">
 <!ENTITY cv-TEMPFILEPREFIX "<envar xmlns='http://www.scons.org/dbxsd/v1.0'>$TEMPFILEPREFIX</envar>">
 <!ENTITY cv-TEMPFILESUFFIX "<envar xmlns='http://www.scons.org/dbxsd/v1.0'>$TEMPFILESUFFIX</envar>">
 <!ENTITY cv-TEX "<envar xmlns='http://www.scons.org/dbxsd/v1.0'>$TEX</envar>">
@@ -1273,6 +1274,7 @@ THIS IS AN AUTOMATICALLY-GENERATED FILE.  DO NOT EDIT.
 <!ENTITY cv-link-TEMPFILEARGESCFUNC "<link linkend='cv-TEMPFILEARGESCFUNC' xmlns='http://www.scons.org/dbxsd/v1.0'><envar>$TEMPFILEARGESCFUNC</envar></link>">
 <!ENTITY cv-link-TEMPFILEARGJOIN "<link linkend='cv-TEMPFILEARGJOIN' xmlns='http://www.scons.org/dbxsd/v1.0'><envar>$TEMPFILEARGJOIN</envar></link>">
 <!ENTITY cv-link-TEMPFILEDIR "<link linkend='cv-TEMPFILEDIR' xmlns='http://www.scons.org/dbxsd/v1.0'><envar>$TEMPFILEDIR</envar></link>">
+<!ENTITY cv-link-TEMPFILEENCODING "<link linkend='cv-TEMPFILEENCODING' xmlns='http://www.scons.org/dbxsd/v1.0'><envar>$TEMPFILEENCODING</envar></link>">
 <!ENTITY cv-link-TEMPFILEPREFIX "<link linkend='cv-TEMPFILEPREFIX' xmlns='http://www.scons.org/dbxsd/v1.0'><envar>$TEMPFILEPREFIX</envar></link>">
 <!ENTITY cv-link-TEMPFILESUFFIX "<link linkend='cv-TEMPFILESUFFIX' xmlns='http://www.scons.org/dbxsd/v1.0'><envar>$TEMPFILESUFFIX</envar></link>">
 <!ENTITY cv-link-TEX "<link linkend='cv-TEX' xmlns='http://www.scons.org/dbxsd/v1.0'><envar>$TEX</envar></link>">