-add copyright

-shorten variable names (for size)
-simplify sed() command scan
-sed backup extension includes the . dot now
-add extended commands in tf_extend.py

README.md

@@ -2,6 +2,8 @@
 
 A module for manipulating files in the *MicroPython* environment.  
 
+[TOC]
+
 ## Oveview
 
 I discovered *MicroPython* when working on the ESP8266 processor. Everything seemed very nice, except it was awkward moving files around. All the methods I could find required a back-and-forth with the programmer's desktop.
@@ -69,7 +71,9 @@ tf.cp('log.txt','log.bak')`
    returns: Null
 ```
 
-Simply copies a source file to a destination file. Filenames may include folders or . or .. prefixes. The destination is overwritten if it exists. This function reads-&-writes one line at a time, so it can handle megabyte files. Typical speeds are 100kB/sec on an ESP8266.
+Simply copies a source file to a destination file. Filenames may include folders or . or .. prefixes. The destination is overwritten if it exists. This function reads-&-writes one line at a time, so it can handle megabyte files. Typical speeds are 100kB/sec on an ESP8266. 
+
+**NOTE** this function *only works on text files*. Line lengths of up to 4096 work fine on the ESP8266.
 
 #### cat()
 
@@ -124,7 +128,7 @@ tf.grep('config.ini', '\[\w*\]', numbers = True)
 #### sed()
 
 ```
-  sed(filename, pattern, bak_ext="bak")
+  sed(filename, pattern, bak_ext=".bak")
   in:  filename       the file to edit
        pattern        a sed pattern, involving one of "aidsxX"
        bak_ext        the extension to use when creating the file backup (without the dot)
@@ -172,7 +176,9 @@ s/thier/their/
 s@ratio\s*=\s*num/denom@ratio = num/denom if denom else 0@
 ```
 
-**Note**: you will need some free space on your disk, the same size as the source file, as a backup file is *always* made. To edit an 800k file, you should have 800k of free space.
+**Note**: The function version of sed() can have embedded space characters in the pattern; the command line version (below) requires single-quotes around patterns that have space characters.
+
+**Note**: You will need some free space on your disk, the same size as the source file, as a backup file is *always* made. To edit an 800k file, you should have 800k of free space.
 
 **Note**: The functions for
 
@@ -224,7 +230,7 @@ grep '^\s*#define\s+[A-Z]' main.c
 sed 1,100s/recieve/receive/ doc.txt
 sed '33-$s/it is/it\'s/' doc.txt
 sed '45i   a new line of indented text' doc.txt
-```    
+```
 
 The **REPL** typing-history is functional, so you can use the up-arrow to recall the last 4-5 commands. Use the left-arrow and backspace to correct lines that have errors.
 
@@ -235,17 +241,18 @@ Commands with invalid syntax return a line of information, and are ignored. Non
 In its present form, the module has these limitations:  
 
 * filenames are limited to 255 chars
-* search patterns involving \ escapes may or may not work properly
-* the esp8266 implementation does not allow \1,\2 type pattern substitution
+* files must be text
+	* or have a `\n` at least every 4096 characters
+	* `sed()` requires lines <=2048 characters, and this `sed()` won't match binary chars
+* search patterns involving \ escapes other than `\'` probably won't work
 * in the simple shell
-    * filenames must not have spaces
-    * patterns with spaces ***must*** be quoted
-    * the target of `cp`and `mv` *cannot* be a simple a directory-name as in Linux; write the whole filename *w.r.t,* the current directory
+  * filenames must not have spaces
+  * patterns with spaces ***must*** be quoted
+  * the target of `cp`and `mv` *cannot* be a simple a directory-name as in Linux; write the whole filename *w.r.t,* the current directory
 * the complexity of pattern matching is limited. 
-    * try to format the grep patterns so they avoid deep stack recursion. For example, '([^#]|\\#)\*' has a very generous search term as the first half, and can cause deep-stack recursion. The equivalent '(\\#|[^#]\*)' is more likely to succeed.
-* with sed-search-and-replace, the parsed line *includes* the terminal \n, so if you replace the text all the way to the end of the line, you will delete the terminal \n and this line will merge with the subsequent line
-    * if the replacement is '', the line will appear to vanish, e.g. `s/^#.*//`  will delete comment lines
-    * pattern matching to \n and \r does not work
+  * try to format the grep patterns so they avoid deep stack recursion. For example, '([^#]|\\#)\*' has a very generous search term as the first half, and can cause deep-stack recursion. The equivalent '(\\#|[^#]\*)' is more likely to succeed.
+* with sed, lines are parsed and saved one-line-at-a-time, so pattern matching to \n and \r does not work
+* this simple shell is different than [mpfshell](https://github.com/wendlers/mpfshell) in that this shell runs entirely on the target device. There is no allowance in this shell for transferring files in/out of the target.
 
 ## Examples
 
@@ -276,14 +283,16 @@ Search a log file for an incident
 [command line]
    grep [Ee]rror log.txt
    grep '2021-02-12 16:\d\d' log.txt
-   [search and keep a record ]
+   # search and keep a record
    cp log.txt log.details
    sed 'x/2021-02-12 16:\d\d` log.details
 ```
 
 ## Installation
 
-Move the 'tf.py' file over to the target. You can use `webrepl` [command line program](https://github.com/micropython/webrepl)  or the **WEBREPL** [web page](https://micropython.org/webrepl/) .
+~If you need help with getting connected to your MicroPython board, there are excellent howto guides here and here~ TODO! 
+
+Move the 'tf.py' file over to the target. You can use `webrepl` [command line program](https://github.com/micropython/webrepl)  or the **WEBREPL** [web page](http://micropython.org/webrepl/) .  If you want the command line extensions, then send over the `tf_extend.py` file as well
 
 Once the module is present in the file system of the target, you can use the **REPL** command line interface to invoke it
 
@@ -315,12 +324,30 @@ This is the *simple command line*. You can type `dir` to get an idea of what's a
  /test_dir$ 
 ```
 
-If you don't need the *simple command line*, you can still use the methods listed above. Feel free to cut the `tf.py` module in half by deleting everything below the line
+If you don't need the *simple command line*, you can still use the methods listed at the top of this readme. Feel free to cut the `tf.py` module in half by deleting everything below the line
+
+```
+  def ext_cmd(a):
+```
+
+## Extensions
+
+I found the simple command line so useful, I added some extra non-file-related functions. These are included in the optional filr `tf_extend.py'.  the available command list is extended to include
 
 ```
-    def help():
+scan                      # scan and show the local AP's
+connect essid password    # create a persistent wifi connection
+ifconfig                  # show current ip address
+host <domain.name>        # do an DNS lookup
+freq [160 | 80]           # get/set the ESP8266 frequency
+exec <python-filename>    # execute a small python file
+free                      # display the heap size: used + free
 ```
 
+The `tf.py` module checks to see if the `tf_extend.py` files exists, and forwards unknown commands to it. The `help` system also extends when the extension file exists.
+
+Installing the extensions module uses about 2k of flash/disk space and 2kB of heap ram.
+
 ## Performance
 
 Typical performance on an ESP8266 @80MHz, 90kB log file, 1200 lines; serial connected terminal @115200baud

tf.py

@@ -1,122 +1,128 @@
 # tf  Text File manipulations
 #   for micropython and other tiny environments 
-
-#NOTE: the ESP8266 port cannot to \1,\2 type replacements in the s/search/replace/ operator
+# (c) Pat Beirne patb@pbeirne.com
 import re,os,sys,gc
 
-def _file_scan(src,dest,start=1,end=0xFFFFFFFF,numbers=False,grep_func=None):
-  #src is a filename, dst is an open handle
+# ====legend====
+# e=end
+# f=input file
+# g=output file
+# i=iteration variable
+# m=line range
+# r=replace text
+# s=start/search/string
+# sp=search regex
+
+def transfer(src,dest,first=1,last=0xFFFFFFFF,numbers=False,grep_func=None):
+  #src is a filename, dst is a handle
   i=0
   try:
     with open(src) as f:
-      for line in f:
+      for lin in f:
         i=i+1
-        if i<start or i>end:
+        if i<first or i>last:
           continue
-        if grep_func and not grep_func(line):
+        if grep_func and not grep_func(lin):
           continue
         if numbers:
           dest.write(str(i)+' ')
-        dest.write(line)
+        dest.write(lin)
   except:
     print("could not open file {}".format(src))
 
-def cp(src_f, dst_f):
+def cp(src,dest):
   try:
-    with open(dst_f,'w') as g:
-      _file_scan(src_f,g)
+    with open(dest,'w') as g:
+      transfer(src,g)
   except:
-    print("could not write to file {}".format(dst_f))
+    print("could not write to file {}".format(dest))
 
 def grep(filename, pattern, numbers=False):
   m=re.compile(pattern)
   if not m:
     print("grep() called with invalid pattern")
-    return None
-  _file_scan(filename,sys.stdout,numbers=numbers,grep_func=(lambda x:m.search(x)))
-  print()
+    return 
+  transfer(filename,sys.stdout,numbers=numbers,grep_func=(lambda x:m.search(x)))
 
 def cat(filename, first=1, last=1000000, numbers=False, title=True):
   if title:
     print("===={}=====".format(filename))
-  _file_scan(filename,sys.stdout,first,last,numbers=numbers)
-  print()
+  transfer(filename,sys.stdout,first,last,numbers=numbers)
 
-def sed(filename, sed_cmd, bak_ext="bak"):
-  #print("sed() called with sed_cmd=<{}>".format(sed_cmd))
+def sed(filename, sed_cmd, bak_ext=".bak"):
   # parse the sed_cmd
-  # group 1,3 are the n-start, n-end    group 4 is command
-  g=re.search("^(\d*)([,-](\d+|\$))?\s*([sdaixX].*)",sed_cmd)
-  if not g:
-    print("sed() failed; 2nd argument must be a number followed by one of sdaixX; no changes applied")
-    return 0,0
-  cmd=g.group(4)
-  #print("sed() cmd parsed into <{}>,<{}> and <{}>".format(g.group(1),g.group(3),g.group(4)))
-
-  start,end=(1,1000000)
-  if g.group(1):
-    start=end=int(g.group(1))
-  if g.group(3):
-    end=1000000 if g.group(3)=='$' else int(g.group(3))
+  # group 1,3 are the n-start, n-end    group 4 is command: aidsxX
+  a=re.search("^(\d*)([,-](\d+|\$))?\s*([sdaixX].*)",sed_cmd)
+  if not a:
+    print("sed() failed; 2nd argument must be a number-range followed by one of sdaixX; no changes applied")
+    return
+  cmd=a.group(4)
+
+  s,e=(1,1000000)
+  if a.group(1):
+    s=e=int(a.group(1))
+  if a.group(3):
+    e=1000000 if a.group(3)=='$' else int(a.group(3))
 
   op=cmd[0]
   if op not in "sdiaxX":
     print("sed requires an operation, one of 's,d,i,a,x or X'")
-    return 0,0
+    return
   #print("sed command parser of <{}> returned {} {} {} {}".format(cmd,sr,de,ins,add))
-  if op in "sxX" and len(cmd)<2: 
-    print("invalid sed argument")
-    return (0,0)
-  if op=='s':
-    dl=cmd[1]
-    gs=re.search("s"+dl+"([^"+dl+"]*)"+dl+"([^"+dl+"]*)"+dl,cmd)
-    if not gs:
-      print("invalid sed search-and-replace pattern")
-      return (0,0)
-    s,r = gs.group(1),gs.group(2)
-    #print("search <{}> and replace <{}>".format(s,r))  
-    sp=re.compile(s) 
-  if op=='X' or op=='x':
+  if op in "sxX":
+    if len(cmd)<2: 
+      print("invalid sed argument")
+      return
     dl=cmd[1]
-    gs=re.search("[xX]"+dl+"([^"+dl+"]*)"+dl,cmd)
+    if op=='s':
+      gs=re.search("s"+dl+"([^"+dl+"]*)"+dl+"([^"+dl+"]*)"+dl,cmd)
+    else:
+      gs=re.search("[xX]"+dl+"([^"+dl+"]*)"+dl,cmd)
     if not gs:
       print("invalid sed search pattern")
-      return (0,0)
-    sp=re.compile(gs.group(1)) 
+      return 0,0
+    if op=='s':
+      ss,r = gs.group(1),gs.group(2)
+      #print("search <{}> and replace <{}>".format(s,r))  
+    else:
+      ss=gs.group(1) 
+      #print("search <{}>".format(s))  
+    sp=re.compile(ss) 
 
-  extra=g.group(4)[1:] + '\n' 
+  extra=a.group(4)[1:] + '\n' 
 
   try:
-    os.rename(filename,filename+'.'+bak_ext)
+    os.rename(filename,filename+bak_ext)
   except:
     print("problem with filename; backup failed; no changes made")
-    return (0,0)
+    return
 
   i=h=0
   try: 
-    with open(filename+'.'+bak_ext) as d:
-      with open(filename,'w') as f:
-        for lin in d:
+    with open(filename+bak_ext) as f:
+      with open(filename,'w') as g:
+        for lin in f:
           i=i+1
-          m=(i>=start and i<=end)
+          m=(i>=s and i<=e)
           if op=='s' and m:
+            lin=lin[:-1]
             if sp.search(lin): h+=1
-            lin=sp.sub(r,lin)
+            lin=sp.sub(r,lin)+'\n'
           if op=='d' and m:
             h+=1
             continue   # delete line
           if op=='i' and m:
             #print("insert a line before {} <{}>".format(i,extra))
-            f.write(extra)
+            g.write(extra)
             h+=1
           if op in "aids":
-            f.write(lin)
-          elif (m and (op=='x' and sp.search(lin)) or (op=='X' and not sp.search(lin))):
-            f.write(lin)
+            g.write(lin)
+          elif m and (op=='x' if sp.search(lin) else op=='X'):
+            g.write(lin)
             h+=1
           if op=='a' and m:
             #print("append a line after {} <{}>".format(i,extra))       
-            f.write(extra)
+            g.write(extra)
             h+=1
         #f.write("--file modifed by sed()--\n")
   except OSError:
@@ -125,31 +131,37 @@ def sed(filename, sed_cmd, bak_ext="bak"):
     print("problem with the regex; try a different pattern")
   return (i, h)
 
-def _dir(d=''):
+def _dir(d='.'):
   try:  
-    for g in os.listdir(d):
-      s=os.stat(d+'/'+g)
-      print("{}rwx all {:9d} {}".format('d' if (s[0] & 0x4000) else '-',s[6],g))
+    for f in os.listdir(d):
+      s=os.stat(d+'/'+f)
+      print("{}rwx all {:9d} {}".format('d' if (s[0] & 0x4000) else '-',s[6],f))
   except:
     print("not a valid directory")
   s=os.statvfs('/')
-  print("disk size:{:8d} KB   disk free: {} KB\n".format(s[0]*s[2]//1024,s[0]*s[3]//1024))
+  print("disk size:{:8d} KB   disk free: {} KB".format(s[0]*s[2]//1024,s[0]*s[3]//1024))
 
+'''-----cut here if you only need the functions-----'''
+def ext_cmd(a):
+  return
+if 'tf_extend.py' in os.listdir():
+  import tf_extend
+  ext_cmd=tf_extend.cmd
 
-'''-----cut here if you only need the above functions-----'''
 def _help():
-  print("simple shell v1.0")
+  print("==Simple shell v1.1")
   print("  cp/copy <src-file> <dest-file>")
-  print("  mv/move <src-file> <dest-file>           rm/del <file>")
-  print("  cd [<folder>]       mkdir <folder>       rmdir <folder>")
+  print("  mv/move <src-file> <dest-file>    \t\trm/del <file>")
+  print("  cd [<folder>]       mkdir <folder>\t\trmdir <folder>")
   print("  dir/ls [<folder>]")
   print("  cat/list [-n] [-l <n>,<m>] <file>")
   print("  grep <pattern> <file>")
   print("  sed <pattern> <file>")
-  print("          where <pattern> is '[<n>,<m>] s/search/replace/' or '<n>[,<m>]d' or '<n>i<text>' or '<n>a<text' ")
-  print("file names must NOT have embedded spaces               options must be early on the command line")
-  print("search patterns with spaces require single-quotes      sed implements s/d/i/a/x/X")
-  print("sed does not work across line boundaries               sed s-patterns: non-/ delimiters are allowed")
+  print("      pattern is '<line-range><op><extra>'   e.g'a/search/replace/', 'x!TODO:!', '43,49d', '8itext'")
+  print("      patterns with spaces require single-quotes   sed ops are one of s/d/i/a/x/X")
+  print("      sed does not work across line boundaries     sed s/x/X-patterns: non-/ delimiters are allowed")
+  print("file names must NOT have embedded spaces           options must be early on the command line")
+  ext_cmd('help')
 
 def parseQuotedArgs(st):
   if st[0]=="'":
@@ -162,23 +174,23 @@ def parseQuotedArgs(st):
     return st.split()[0]
 
 def main():
-  print("simple shell: cp/copy mv/move rm/del cat/list cd dir/ls mkdir rmdir grep sed help")
+  print("Simple shell: cp/copy mv/move rm/del cat/list cd dir/ls mkdir rmdir grep sed help")
   while 1:
     numbers=False
     r=input(os.getcwd()+"$ ")
     rp=r.split()
     if not len(rp): continue
     op=rp[0]
-    if op=='dir' or op=='ls':
-      _dir(rp[1] if len(rp)>1 else '')
-    elif op=='cat' or op=='list':
+    if op in ('dir','ls'):
+      _dir(rp[1] if len(rp)>1 else '.')
+    elif op in ('cat','list'):
       n=(" -n " in r) #print line-nums
-      s,e=(1,1000000) #start/end
-      g=re.search("\s+(-l\s*(\d+)([-,](\d+|\$)?)?)\s+",r[3:])
+      s,e=1,1000000 #start/end
+      g=re.search("\s(-l\s*(\d+)([-,](\d+|\$)?)?)\s+",r[3:])
       if g:
         s=e=int(g.group(2))
         if g.group(3):
-	      e=int(g.group(4)) if g.group(4) and g.group(4).isdigit() else 1000000
+          e=int(g.group(4)) if g.group(4) and g.group(4).isdigit() else 1000000
       cat(rp[-1],s,e,numbers=n)
     elif op=='grep':
       if len(rp)<3:
@@ -189,28 +201,32 @@ def main():
       if len(rp)<3:
         print("sed pattern filename")
         continue
-      lines, hits = sed(rp[-1],parseQuotedArgs(r[4:]))
-      print("Lines processed: {}  Lines modifed: {}".format(lines, hits))
+      r=sed(rp[-1],parseQuotedArgs(r[4:]))
+      if r:
+        print("Lines processed: {}  Lines modifed: {}".format(*r))
     elif op=='cd':
       os.chdir(rp[1] if len(rp)>1 else '/')
     elif op=='help':
       _help()
+      ext_cmd(rp)
+    elif ext_cmd(rp):
+      pass
     else:
       try:
-        if op=='cp' or op=='copy':
+        if op in ('cp','copy'):
           cp(rp[1],rp[2])
         elif op=='mkdir':
           os.mkdir(rp[1])
         elif op=='rmdir':
           os.rmdir(rp[1])
-        elif op=='mv' or op=='move':
+        elif op in('mv','move'):
           os.rename(rp[1],rp[2])
-        elif op=='rm' or op=='del':
+        elif op in('rm','del'):
           os.remove(rp[1])
         else:
           print("command not implemented")
       except IndexError:
-        print("not enough argments; check syntax")
+        print("not enough argments; check syntax with 'help'")
       except OSError:
         print("file not found")
     gc.collect()

tf_extend.py

@@ -0,0 +1,62 @@
+import os,network,socket,time,machine,gc
+def cmd(args):
+  if args[0]=='ifconfig':
+    ifc=network.WLAN().ifconfig()
+    print("IP: {}\tmask: {}\tgateway: {}\tDNS: {}".format(*ifc))
+    return True
+  elif args[0]=='host':
+    if len(args)<2:
+      print("syntax: host <domain.name>")
+      return False
+    print("host <{}> is at {}".format(args[1],socket.getaddrinfo(args[1],80)[0][-1][0]))
+    return True
+  elif args[0]=='connect':
+    if len(args)<3:
+      print("syntax: connect <ssid> <password>")
+      return False
+    w=network.WLAN(network.STA_IF)
+    w.connect(args[1],args[2])
+    print("connecting...",end=' ')
+    time.sleep(3)
+    print(w.ifconfig() if w.isconnected() else "not yet connected; try 'ifconfig' in a few seconds")
+    return True
+  elif args[0]=='scan':
+    w=network.WLAN(network.STA_IF)
+    print("scanning...")
+    s=w.scan()
+    if len(s)==0:
+      print("no AP found")
+      return True
+    for i in s:
+      print("ch: {}\tRSSI: {}\t{}\tSSID: {}".format(i[2],i[3],"open" if i[4]==0 else "",i[0]))
+    return True
+  elif args[0]=='freq':
+    if len(args)==1 or args[1] in ("160","80"):
+      if len(args)>1:
+        machine.freq(int(args[1])*1000000)
+      print("master cpu frequency {}MHz".format(machine.freq()//1000000))
+    else:
+      print("syntax: freq [ 160 | 80 ]")
+    return True
+  elif args[0]=='exec':
+    if len(args)<2:
+      print("syntax: exec <python-filename>")
+    else:
+      try:
+        exec(open(args[1]).read(),globals(),globals())
+      except OSError:
+        print("file not found")
+    return True
+  elif args[0]=='free':
+    print("memory used: {}\tmemory free:{}".format(gc.mem_alloc(),gc.mem_free()))
+    return True
+  elif args[0]=='help':
+    print("==Extended commands")
+    print("  connect <essid> <password> \tscan")
+    print("  ifconfig                   \thost <domain.name>")
+    print("  freq [ 160 | 80 ]          \texec <python-filename>")
+    print("  free")
+    return True
+  else:
+    return False
+