dirnotes-cli is working pretty well; copy the library code from -cli to the other two

README.md

@@ -14,20 +14,30 @@ Table of Contents
 
 ## SYNOPSIS
 
-The **dirnotes** family of apps allows you to add a descriptive comment to a file. The descriptions are stored in two places:
+The **dirnotes** family of apps allows you to add a descriptive comment to a file. 
+The descriptions are stored in two places:
 
  * in the _xattr_ properties of the file
  * in a _database_ located in the user's home directory
 
 [The [MacOS](#macos) stores its comments in a similar way.]
 
-The <code>**dirnotes**</code> app is a GUI app, using the Qt5 framework. At startup, it displays the contents of the current directory, and the comments associated with any of the files or directories. 
-Simple mouse clicks allow you to tunnel down into directories, or rise up the file system. 
-You can create/edit comments and choose whether the _xattr_ or _database_ version of the comments take priority, 
+The <code>**dirnotes**</code> app is a GUI app, using the Qt5 framework. 
+At startup, it displays the contents of the current directory, and the 
+comments associated with any of the files or directories. 
+Simple mouse clicks allow you to add or edit comments, tunnel down 
+into directories, or rise up the file system. 
+You can copy or move files (_with_ comments), and 
+choose whether the _xattr_ or _database_ version of the comments 
+have display priority. 
 
-The <code>**dirnotes-tui**</code> is a very similar app, but uses the _curses_ framework to display its activity in a terminal window. This can be handy if you have to work across a network, or if terminal apps are you preference.
+The <code>**dirnotes-tui**</code> is a very similar app, but uses the 
+_curses_ framework to display its activity in a terminal window. 
+This can be handy if you have to work across a network, 
+or if terminal apps are your preference.
 
-The <code>**dirnotes-cli**</code> is a command line tool, which may be handy for scripting.
+The <code>**dirnotes-cli**</code> is a command line tool, 
+which may be handy for scripting. This all can also do maintenance on the database.
 
 ## USAGE
 
@@ -40,38 +50,47 @@ _s_ for sort, _M_ to change between xattr/database priority.
 
 The **<code>dirnotes-cli</code>** has options for _-l_ list and _-c_ create a comment. 
 
-All three apps in the **dirnotes** family have the ability to copy files from the current directory. 
+All three apps in the **dirnotes** family have the ability to 
+copy/move files from the current directory, keeping the comments intact. 
+All three apps have the **-h** option which shows command line usage. 
  
 ## INSTALLATION
 
 Each of the 3 apps in the family is self contained. 
 The <code>**dirnotes**</code> app requires _Python3_ and the _Qt5_ framework. 
-The <code>**dirnotes-tui**</code> and <code>**dirnotes-cli**</code> apps simply require _Python3_.
+The <code>**dirnotes-tui**</code> and <code>**dirnotes-cli**</code> apps 
+simply require _Python3_.
+
+Simply copy the file into your path, to ~/.local/bin for example. 
+For a better GUI experience, copy 
+<code>dirnotes.desktop</code> to <code>~/.local/share/applications</code> and 
+<code>dirnotes.xpm</code> to <code>~/.local/share/icons/</code> 
 
 ### CONFIG FILE
 
-By default, the file **~/.dirnotes.conf** will be used to load the user's config. 
+By default, the file **~/.config/dirnotes/dirnotes.conf** will be used to 
+load the user's config. 
 This is a JSON file, with three attributes that are important:
 
-> * xattr_tag (default: <code>usr.xdg.comment</code>)
-> * database (default: <code>~/.dirnotes.db</code>, sensible alt: <code>/var/lib/dirnotes.db</code>) 
-> * start_mode (_xattr_ or _db_ priority)
+> * xattr_tag (default: <code>usrr.xdg.comment</code>)
+> * database (default: <code>~/.local/share/dirnotes/dirnotes.db</code>, sensible alt: <code>/var/lib/dirnotes.db</code>) 
+> * start_mode (_xattr_ or _db_ display priority)
 
 The _config_file_ should be auto-generated the first time one of the **dirnotes** apps is run.
-[_not fully implemented_]
 
 ## LIMITATIONS 
 
-
 The file comments are located in two locations: a database, and in the 
 xattr properties of the file. Each of these storage locations has its 
 own benefits and limitations. These can be summed up: **_xattr_** comments
 follow the iNode, **_database_** comments follow the file name.
 
+
 ### xattr
 
 Comments stored in the xattr properties can be copied/moved with the file, if you
-use the correct options: <code>**cp -p**</code>. The <code>**mv**</code> utility 
+use the correct options: <code>**cp&nbsp;-p&nbsp;_src&nbsp;dest_**</code>. 
+The <code>**mv**</code> utility 
 automatically preserves _xattr_. Other programs can also be coerced into 
 perserving _xattr_ properties:
 
@@ -81,8 +100,11 @@ perserving _xattr_ properties:
 
 Not all file systems support xattr properties (vfat/exfat does not).
 
+_xattr_ comments may only be applied to files for which the user has _write_ permission.
+
 The current implementation of <code>**sshfs**</code> and <code>**scp**</code> 
-do not support the copy of _xattr_ properties.
+do not support copying of _xattr_ properties. **Dropbox** type mounts are 
+unlikely to support _xattr_ comments.
 If you want to copy files to a remote machine and include the _xattr_ comments, use <code>**rsync**</code> with the _-X_ option. Or <code>**tar**</code>.
 
 Some editing apps (like _vim_) will create a new file when saving the data, which orphans the _xattr_ comments. For these apps, use the _database_ system.  
@@ -91,23 +113,30 @@ Removable disk devices (usb sticks) which are formatted with a Linux-based files
 
 ## database
 
-Comments stored in the database work for all filesystem types (including vfat/exfat/sshfs)
+Comments stored in the database work for all filesystem types 
+(including vfat/exfat/sshfs)
  
-Comments are personalized to the _current user_.
-Another user on the same system will not see these comments.
+The _database_ comments that are stored in 
+<code>~/.local/share/dirnotes/dirnotes.db</code> are inherently associated 
+with a single user. If the _database_ is located in 
+<code>/var/lib/dirnotes.db</code>, it can be shared by all the users in the system.
 
 Files are indexed by their complete path name. Removable filesystems should be
-mounted in a consistent way, so that the complete path name is reproducable.
+mounted in a consistent way, so that the complete path name is reproducable. 
+Symlinks are _not_
+dereferenced, so they may have comments bound to them.
 
 Comments stored in the database do not travel with the files when
 they are moved or copied, unless using the **dirnotes** family of tools. 
 
-The _database_ comments that are stored in <code>~/.dirnotes.db</code> are inherently associated with a single user. If the _database_ is located in <code>/var/lib/dirnotes.db</code>, it is shared by all the users in the system. The comment with the 'most recent timestamp' wins.
+_Database_ comments may be applied to any visible file, _even if they are readonly_. 
+For exmple, comments may be attached to the files in <code>/usr/bin/\*</code> even though they are probably owned by _root_.
 
 ## PROGRAMMER NOTES
 
 Instead of an API, here is how you can get directly at the underlying comment data. 
-If you intend to use the **dirnotes** apps, try to keep the two versions of the comments in sync.
+If you intend to use the **dirnotes** apps, 
+try to keep the two versions of the comments in sync.
 
   * xattr
 
@@ -122,11 +151,14 @@ to display the comments/author/date on a file. For example:
         user.xdg.comment.author: patb
         user.xdg.comment.date: 2022-09-29 08:07:42
 
-The other options on the **xattr** command line tool allow you to write (*xattr -w*) or delete (*xattr -d*) the comments.
+The other options on the **xattr** command line tool allow you to 
+write (*xattr -w*) or delete (*xattr -d*) the comments.
 
   * database
 
-The comments are stored in an Sqlite3 database, usually located at "~/.dirnotes.db". The database itself is contained within that file, and its schema is this:
+The comments are stored in an Sqlite3 database, usually 
+located at "~/.local/share/dirnotes/dirnotes.db". 
+The database itself is contained within that file, and its schema is this:
 
 ~~~~
     CREATE TABLE dirnotes (name TEXT, date DATETIME, size INTEGER, comment TEXT, comment_date DATETIME, author TEXT)
@@ -143,9 +175,12 @@ The comments are stored in an Sqlite3 database, usually located at "~/.dirnotes.
 |author     |   the system name of the user who created the comment  |  patb | 
 
 
-The _date_ and _size_ fields reflect the file's modification date and size at the time of the last edit of the file comment, which is stored in _comment_date_.
+The _date_ and _size_ fields reflect the file's modification date and size 
+at the time of the last edit of the file comment, which is stored in _comment_date_.
 
-As comments are editted or appended, new records are added to the database. Older records are are not purged. This gives you a history of the comments, but it means that fetching the most recent comment involves something like
+As comments are editted or appended, new records are added to the database. 
+Older records are are not purged. This gives you a history of the comments, 
+but it means that fetching the most recent comment involves something like
 
 ~~~~
   SELECT * FROM dirnotes WHERE name=? ORDER BY comment_date DESC
@@ -157,10 +192,24 @@ The database is created the first time one of the **dirnotes** apps is run.
 
   * misc
 
-The <code>**dirnotes**</code> gui app has a desktop icon built into the code. There is not need for an external .icon file.
+The <code>**dirnotes**</code> gui app has a desktop icon built into the code. 
+There is not need for an external .icon file, but there is an .xpm file included
+in the project, which can be copied to ~/.local/share/icons/
+
+### comment date & author
+
+The <code>copy()/move()</code> methods that are built into the **dirnotes** library
+will ask the operating system to copy/move the file _with_ xattr intact. 
+The entry in the database is created _at the time of invocation_. 
+Therefore, the xattrs will reflect the original author+date on the comments, 
+whereas the database version is updated on each copy/move; 
+the dirnotes-comments details will therefor diverge over time.
 
 There was _no_ consideration given for language translation. Email [me](mail:patb@pbeirne.com) if you want this, or can help.
 
+All these apps only accomadate a single line comment. An embedded newline will 
+cause unpredictable behaviour. 
+
 ### MacOS {#macos}
 
 The **MacOS** inherently supports file comments. The Finder app manages most of the user activity. It handles file comments in a similar manner to **Dirnotes**. Comments are stored in two places:
@@ -192,4 +241,18 @@ the three apps. And there _may_ be some inconsistency.
   The _qt-gui_ app is working pretty well.  
 
 
+QUESTIONS:
+
+There are several open-ended questions that need to be answered. 
+Does anyone have an opinion?
+
+1. How important is multi-line comments?
+
+2. Is it ok to put the config file and database file buried in ~/.config and ~/.local? 
+
+These directories exist on computers with a gui/windowing system installed, but don't neccessarily exist on headless servers. Perhaps the default locations should be in the user directory? (~/.dirnotes.conf and ~/.dirnotes.db)
+
+3. Who needs translations?
+
+4. Does anybody have a better edit-window for CURSES?
 

dirnotes

@@ -21,7 +21,7 @@ nav tools are enabled, so you can double-click to go into a dir
 
 """
 
-VERSION = "0.7"
+VERSION = "0.8"
 
 helpMsg = f"""<table width=100%><tr><td><h1>Dirnotes</h1><td>
 <td align=right>Version: {VERSION}</td></tr></table>
@@ -61,7 +61,7 @@ the comment box gets a light yellow background.
 double-click the mouse, or hit &lt;Enter&gt;.
 """
 
-import sys,os,time,argparse,stat,getpass,shutil,logging
+import sys,os,time,argparse,stat,getpass,shutil,logging,math
 from   PyQt5.QtGui     import *
 from   PyQt5.QtWidgets import *
 from   PyQt5.QtCore    import Qt, pyqtSignal
@@ -80,6 +80,13 @@ mode       = "db"
 
 global mainWindow, dbName
 
+verbose = None
+def print_v(*a):
+  if verbose:
+    print(*a)
+
+############# the DnDataBase, UiHelper and FileObj code is shared with other dirnotes programs
+
 DEFAULT_CONFIG_FILE = "~/.dirnotes.conf"
 
 # config
@@ -92,13 +99,28 @@ DEFAULT_CONFIG = {"xattr_tag":"user.xdg.comment",
   "options for start_mode":("db","xattr")
 }
 
-verbose = None
-def print_v(*a):
-  if verbose:
-    print(*a)
+class ConfigLoader:    # singleton
+  def __init__(self, configFile):
+    configFile = os.path.expanduser(configFile)
+    try:
+      with open(configFile,"r") as f:
+        config = json.load(f)
+    except json.JSONDecodeError:
+      errorBox(f"problem reading config file {configFile}; check the JSON syntax")
+      config = DEFAULT_CONFIG
+    except FileNotFoundError:
+      errorBox(f"config file {configFile} not found; using the default settings")
+      config = DEFAULT_CONFIG
+      try:
+        with open(configFile,"w") as f:
+          json.dump(config,f,indent=4)
+      except:
+        errorBox(f"problem creating the config file {configFile}")
+    self.dbName = os.path.expanduser(config["database"])
+    self.mode = config["start_mode"]    # can get over-ruled by the command line options
+    self.xattr_comment = config["xattr_tag"]
 
-############# the dnDataBase and FileObj code is shared with other dirnotes programs
-class dnDataBase:
+class DnDataBase:
   ''' the database is flat
     fileName: fully qualified name
     st_mtime: a float
@@ -137,6 +159,7 @@ class dnDataBase:
     except sqlite3.OperationalError:
       self.writable = False
 
+class UiHelper:
   @staticmethod
   def epochToDb(epoch):
     return time.strftime(DATE_FORMAT,time.localtime(epoch))
@@ -148,15 +171,26 @@ class dnDataBase:
     now = time.time()
     diff = now - longDate
     if diff > YEAR:
-      fmt = "%b %e %Y"
+      fmt = "%b %e  %Y"
     else:
       fmt = "%b %e %H:%M"
     return time.strftime(fmt, time.localtime(longDate))
-    
+  @staticmethod
+  def getShortSize(fo):
+    if fo.isDir():
+      return " <DIR> "
+    elif fo.isLink():
+      return " <LINK>"
+    size = fo.getSize()
+    log = int((math.log10(size+1)-2)/3)
+    s = " KMGTE"[log]
+    base = int(size/math.pow(10,log*3))
+    return f"{base}{s}".strip().rjust(7)
+
 
 ## one for each file
 ## and a special one for ".." parent directory
-class FileObj():
+class FileObj:
   """  The FileObj knows about both kinds of comments. """
   def __init__(self, fileName, db):
     self.fileName = os.path.abspath(fileName)     # full path; dirs end WITHOUT a terminal /
@@ -166,16 +200,18 @@ class FileObj():
       if not self.displayName.endswith('/'):
         self.displayName += '/'
     self.date = self.stat.st_mtime
-    self.size = self.stat.st_size
+    self.size = self.stat.st_size 
     self.db = db
 
   def getName(self):
+    """ returns the absolute pathname """
     return self.fileName
   def getDisplayName(self):
+    """ returns just this basename of the file; dirs end in / """
     return self.displayName
 
-  # new methods; no caching
   def getDbData(self):
+    """ returns (comment, author, comment_date) """
     if not hasattr(self,'dbCommentAuthorDate'):
       cad = self.db.execute("select comment, author, comment_date from dirnotes where name=? order by comment_date desc",(self.fileName,)).fetchone()
       self.dbCommentAuthorDate = cad if cad else (None, None, None)
@@ -184,6 +220,7 @@ class FileObj():
     return self.getDbData()[0]
 
   def getXattrData(self):
+    """ returns (comment, author, comment_date) """
     if not hasattr(self,'xattrCommentAuthorDate'):
       c = a = d = None
       try:
@@ -209,7 +246,7 @@ class FileObj():
           (self.fileName, s.st_mtime, s.st_size,
           str(newComment), time.time(), getpass.getuser()))
       self.db.commit()
-      self.dbCommentAuthorDate = newComment, getpass.getuser(), dnDataBase.epochToDb(time.time())
+      self.dbCommentAuthorDate = newComment, getpass.getuser(), UiHelper.epochToDb(time.time())
     except sqlite3.OperationalError:
       print_v("database is locked or unwritable")
       errorBox("the database that stores comments is locked or unwritable")
@@ -235,13 +272,15 @@ class FileObj():
         errorBox("is this a VFAT or EXFAT volume? these don't allow comments")
       return False
 
-  def getComment(self):
+  def getComment(self,mode):
     return self.getDbComment() if mode == "db"    else self.getXattrComment()
-  def getOtherComment(self):
+  def getOtherComment(self,mode):
     return self.getDbComment() if mode == "xattr" else self.getXattrComment()
-  def getData(self):
+  def getData(self,mode):
+    """ returns (comment, author, comment_date) """
     return self.getDbData()    if mode == "db"    else self.getXattrData()
-  def getOtherData(self):
+  def getOtherData(self,mode):
+    """ returns (comment, author, comment_date) """
     return self.getDbData()    if mode == "xattr" else self.getXattrData()
 
   def getDate(self):
@@ -258,16 +297,30 @@ class FileObj():
   def copyFile(self, destDir):
     # NOTE: this method copies the xattr (comment + old author + old date) 
     #       but creates new db (comment + this author + new date)
-    if stat.S_ISREG(self.stat.st_mode):
-      dest = os.path.join(destDir,self.displayName)
-      try:
-        print("try copy from",self.fileName,"to",dest)
-        shutil.copy2(self.fileName, dest)
-      except:
-        errorBox(f"file copy to <{dest}> failed; check permissions")
-      f = FileObj(dest, self.db)
-      print("dest object created",repr(f))
-      f.setDbComment(self.getDbComment())
+    dest = os.path.join(destDir,self.displayName)
+    try:
+      print_v("try copy from",self.fileName,"to",dest)
+      shutil.copy2(self.fileName, dest)
+    except:
+      errorBox(f"file copy to <{dest}> failed; check permissions")
+      return
+    f = FileObj(dest, self.db)
+    f.setDbComment(self.getDbComment())
+  def moveFile(self, destDir):
+    # NOTE: this method moves the xattr (comment + old author + old date)
+    #       but creates new db (comment + this author + new date)
+    src  = self.fileName
+    dest = os.path.join(destDir, self.displayName)
+    # move preserves dates & chmod/chown & xattr
+    print_v(f"move from {self.fileName} to {destDir}")
+    try:
+      shutil.move(src, dest)
+    except:
+      ErrorBox(f"file move to <{dest}> failed; check permissions")
+      return
+    # and copy the database record
+    f = FileObj(dest,self.db)
+    f.setDbComment(self.getDbComment())  
 
 class HelpWidget(QDialog):
   def __init__(self, parent):
@@ -431,7 +484,7 @@ class DirNotes(QMainWindow):
     mf.addSeparator()
     mf.addAction("Change mode", self.switchMode, "Ctrl+M")
     mf.addAction("Copy file", self.copyFile, "Alt+C")
-    mf.addAction("Move file", self.moveFile, "Alt+V")
+    mf.addAction("Move file", self.moveFile, "Alt+M")
     mf.addSeparator()
     mf.addAction("Quit", self.close, QKeySequence.Quit)
     mf.addAction("About", self.about, QKeySequence.HelpContents)
@@ -475,6 +528,9 @@ class DirNotes(QMainWindow):
     lb.setHorizontalHeaderItem(3,QTableWidgetItem("Comment"))
     lb.setSortingEnabled(True)
 
+    self.sameBrush   = QBrush(QColor(255,255,255))
+    self.differBrush = QBrush(QColor(255,255,160))
+
     self.refill()
     lb.resizeColumnsToContents()
     
@@ -521,27 +577,29 @@ class DirNotes(QMainWindow):
     #self.lb.superKeyEvent(e)
     super().keyPressEvent(e)
 
-  def copyFile(self):
+  def copyMoveFile(self, doCopy, pickerTitle):
     # get current selection
     r, c = self.lb.currentRow(), self.lb.currentColumn()
     fo = self.lb.item(r,c).file_object
     if not fo.isDir() and not fo.isLink() and not fo.isSock(): 
-      print_v(f"copy file {fo.getName()}")
+      print_v(f"{'copy' if doCopy=='copy' else 'move'} file {fo.getName()}")
       # open the dir.picker
-      r = QFileDialog.getExistingDirectory(self.parent, "Select destination for FileCopy")
-      if r:
-        print_v(f"copy to {r}")
-        fo.copyFile(r)
+      d = QFileDialog.getExistingDirectory(self.parent, pickerTitle)
+      if d:
+        print_v(f"senf file to {d}")
+        fo.copyFile(d) if doCopy=='copy' else fo.moveFile(d)
+  def copyFile(self):
+    self.copyMoveFile('copy',"Select destination for FileCopy")
   def moveFile(self):
-    # TODO: write this
-    print_v("moveFile not yet written")
-      
+    self.copyMoveFile('move',"Select destination for FileMove")
+    self.refill()
+
   def refill(self):
     self.refilling = True
     self.lb.sortingEnabled = False
     self.directory = FileObj(self.curPath,self.db)
     
-    self.thisDirLabel.setText(f'<table width=100%><tr><th><b>{self.directory.getDisplayName()}</b></th><th style"text-align:right;">{self.directory.getComment()}</th></tr></table>')
+    self.thisDirLabel.setText(f'<table width=100%><tr><th><b>{self.directory.getDisplayName()}</b></th><th style"text-align:right;">{self.directory.getComment(mode)}</th></tr></table>')
     (self.modeShow.setText("<i>Showing comments stored in extended attributes</i><br>(xattr: user.xdg.comment)") 
       if mode=="xattr" else 
       self.modeShow.setText("<i>Showing comments from the database</i><br>(~/.dirnotes.db)"))
@@ -579,36 +637,36 @@ class DirNotes(QMainWindow):
       self.lb.setItem(i,0,item)
 
       # get the comment from database & xattrs, either can fail
-      comment, auth, cdate = this_file.getData()
-      other_comment = this_file.getOtherComment()
+      comment, auth, cdate = this_file.getData(mode)
+      other_comment = this_file.getOtherComment(mode)
       ci = SortableTableWidgetItem(comment,comment or '~',this_file)
       ci.setToolTip(f"comment: {comment}\ncomment date: {cdate}\nauthor: {auth}")
       if other_comment != comment:
-        ci.setBackground(QBrush(QColor(255,255,160)))
+        ci.setBackground(self.differBrush)
         print_v("got differing comments <{}> and <{}>".format(comment, other_comment))
       ci.setFlags(Qt.ItemIsSelectable | Qt.ItemIsEditable | Qt.ItemIsEnabled)
       self.lb.setItem(i,3,ci)
 
       dt = this_file.getDate()
-      da = SortableTableWidgetItem(dnDataBase.getShortDate(dt),dt,this_file)
+      da = SortableTableWidgetItem(UiHelper.getShortDate(dt),dt,this_file)
       da.setToolTip(time.strftime(DATE_FORMAT,time.localtime(dt)))
       da.setFlags(Qt.ItemIsEnabled)
       self.lb.setItem(i,1,da)
 
       si = this_file.getSize()
       if this_file.isDir():
-        sa = SortableTableWidgetItem('',0,this_file)
+        sa = SortableTableWidgetItem(UiHelper.getShortSize(this_file),0,this_file)
         sa.setIcon(dirIcon)
       elif this_file.isLink():
-        sa = SortableTableWidgetItem('symlink',-1,this_file)
+        sa = SortableTableWidgetItem(UiHelper.getShortSize(this_file),-1,this_file)
         sa.setIcon(linkIcon)
         dst = os.path.realpath(this_file.getName())
         sa.setToolTip(f"symlink: {dst}")
       elif this_file.isSock():
-        sa = SortableTableWidgetItem('socket',-1,this_file)
+        sa = SortableTableWidgetItem(UiHelper.getShortSize(this_file),-1,this_file)
         sa.setIcon(sockIcon)
       else:
-        sa = SortableTableWidgetItem(str(si),si,this_file)
+        sa = SortableTableWidgetItem(UiHelper.getShortSize(this_file),si,this_file)
         sa.setIcon(fileIcon)
       sa.setTextAlignment(Qt.AlignRight)
       sa.setFlags(Qt.ItemIsEnabled)
@@ -622,16 +680,19 @@ class DirNotes(QMainWindow):
   def change(self,x):
     if self.refilling:
       return
-    print_v("debugging " + x.text() + " r:" + str(x.row()) + " c:" + str(x.column()))
-    print_v("      selected file: "+self.lb.item(x.row(),0).file_object.getName())
     the_file = self.lb.item(x.row(),0).file_object
-    print_v("      and the row file is "+the_file.getName())
+    print_v(f"debugging {x.text()} r:{str(x.row())} c:{str(x.column())}")
+    print_v(f"      selected file: {the_file.getName()}")
     the_file.setDbComment(str(x.text()))
     the_file.setXattrComment(str(x.text())) 
-    if the_file.getComment() == the_file.getOtherComment():
-      x.setBackground(QBrush(QColor(255,255,255)))
+
+    # set the background (wrap it, because of reentry to .change())
+    self.refilling = True 
+    if the_file.getComment(mode) == the_file.getOtherComment(mode):
+      x.setBackground(self.sameBrush)
     else:
-      x.setBackground(QBrush(QColor(255,255,160)))
+      x.setBackground(self.differBrush)
+    self.refilling = False
 
   def switchMode(self):
     global mode
@@ -641,27 +702,6 @@ class DirNotes(QMainWindow):
     self.lb.setCurrentCell(row,column)
     self.lb.setFocus(True)
 
-  # TODO: this may not be needed
-  def restore_from_database(self):
-    print("restore from database")
-    # retrieve the full path name
-    fileName = str(self.lb.item(self.lb.currentRow(),0).file_object.getName())
-    print("using filename: "+fileName)
-    existing_comment = str(self.lb.item(self.lb.currentRow(),3).text())
-    print("restore....existing="+existing_comment+"=")
-    if len(existing_comment) > 0:
-      m = QMessageBox() 
-      m.setText("This file already has a comment. Overwrite?")
-      m.setStandardButtons(QMessageBox.Ok | QMessageBox.Cancel);
-      if m.exec_() != QMessageBox.Ok:
-        return
-    fo_row = self.db.getData(fileName)
-    if fo_row and len(fo_row)>1:
-      comment = fo_row[3]
-      print(fileName,fo_row[0],comment) 
-      the_file = dn.files[self.lb.currentRow()]
-      the_file.setComment(comment)
-      self.lb.setItem(self.lb.currentRow(),3,QTableWidgetItem(comment))
     
 def parse():
   parser = argparse.ArgumentParser(description='dirnotes application')
@@ -691,32 +731,16 @@ if __name__=="__main__":
   print_v(f"using {p.dirname}")
   verbose = p.verbose
   
-  config_file = p.config_file if p.config_file else DEFAULT_CONFIG_FILE
-  config_file = os.path.expanduser(config_file)
-  config = DEFAULT_CONFIG
-  try:
-    with open(config_file,"r") as f:
-      config = json.load(f)
-  except json.JSONDecodeError:
-    print(f"problem reading config file {config_file}; check the .json syntax")
-  except FileNotFoundError:
-    print(f"config file {config_file} not found, using the default settings and writing a default")
-    try:
-      with open(config_file,"w") as f:
-        json.dump(config,f,indent=4)
-    except:
-      print(f"problem creating the file {config_file}")
+  config = ConfigLoader(p.config_file or DEFAULT_CONFIG_FILE)
   
   print_v(f"here is the .json {repr(config)}")
-  dbName = os.path.expanduser(config["database"])
-  db = dnDataBase(dbName).db
-  xattr_comment = config["xattr_tag"]
+  dbName = config.dbName 
+  db = DnDataBase(dbName).db
+  xattr_comment = config.xattr_comment
   xattr_author  = xattr_comment + ".author"
   xattr_date    = xattr_comment + ".date"
 
-  mode = "db"
-  if "start_mode" in config and config["start_mode"] in modes:
-    mode = config["start_mode"] 
+  mode = config.mode 
   if p.xattr:
     mode = "xattr" 
   if p.db:

dirnotes-cli

@@ -1,120 +1,372 @@
 #!/usr/bin/python3
-# TODO starting with a dir shoild list all-files
-# TODO: get rid of the -n option; multiple files are prefixed by 'filename:', single files aren't
+# TODO: option to print out full path name; most useful in the .json output format
 
-VERSION = "0.3"
+VERSION = "0.4"
 
 import os, sys, argparse, xattr, json, sqlite3
 
+# global mutables 
 answer_json = []
-verbose = 0
+verbose = debug = 0
 db = None
 xattr_comment = "user.xdg.comment"
-xattr_author    = "user.xdg.comment.author"
-xattr_date      = "user.xdg.comment.date"
+xattr_author  = "user.xdg.comment.author"
+xattr_date    = "user.xdg.comment.date"
+mode          = "db"
 
 #======= debugging/verbose ===========
 def print_d(*a):
-    if verbose > 1:
-        print('>>',*a)
-def print_v(*a):
-    if verbose:
-        print('>',*a)
+    if debug:
+        print('>>', *a)
+def errorBox(*a):
+    print(*a)
+
+# >>> snip here <<<
+#============ the DnDataBase, UiHelper and FileObj code is shared with other dirnotes programs
+import getpass, time, stat, shutil
+
+DEFAULT_CONFIG_FILE = "~/.config/dirnotes/dirnotes.conf" # or /etc/dirnotes.conf
+
+# config
+#    we could store the config in the database, in a second table
+#    or in a .json file
+DEFAULT_CONFIG = {"xattr_tag":"user.xdg.comment",
+  "database":"~/.local/share/dirnotes/dirnotes.db",
+  "start_mode":"xattr",
+  "options for database":("~/.local/share/dirnotes/dirnotes.db","~/.dirnotes.db","/etc/dirnotes.db"),
+  "options for start_mode":("db","xattr")
+}
+
+class ConfigLoader:    # singleton
+  def __init__(self, configFile):
+    configFile = os.path.expanduser(configFile)
+    try:
+      with open(configFile,"r") as f:
+        config = json.load(f)
+    except json.JSONDecodeError:
+      errorBox(f"problem reading config file {configFile}; check the JSON syntax")
+      config = DEFAULT_CONFIG
+    except FileNotFoundError:
+      errorBox(f"config file {configFile} not found; using the default settings")
+      config = DEFAULT_CONFIG
+      try:
+        os.makedirs(os.path.dirname(configFile),exist_ok = True)
+        with open(configFile,"w") as f:
+          json.dump(config,f,indent=4)
+      except:
+        errorBox(f"problem creating the config file {configFile}")
+    self.dbName = os.path.expanduser(config["database"])
+    self.mode = config["start_mode"]    # can get over-ruled by the command line options
+    self.xattr_comment = config["xattr_tag"]
+
+class DnDataBase:
+  ''' the database is flat
+    fileName: fully qualified name
+    st_mtime: a float
+    size: a long
+    comment: a string
+    comment_time: a float, the time of the comment save
+    author: the username that created the comment
+
+    this object: 1) finds or creates the database
+      2) determine if it's readonly
+
+    TODO: the database is usually associated with a user, in $XDG_DATA_HOME (~/.local/share/)
+    TODO: if the database is not found, create it in $XDG_DATA_DIRS (/usr/local/share/)
+      make it 0666 permissions (rw-rw-rw-)
+  '''
+  def __init__(self,dbFile):
+    '''try to open the database; if not found, create it'''
+    try:
+      self.db = sqlite3.connect(dbFile)
+    except sqlite3.OperationalError:
+      print_d(f"Database {dbFile} not found")
+      try: 
+        os.makedirs(os.path.dirname(dbFile), exist_ok = True)
+        self.db = sqlite3.connect(dbFile)
+      except (sqlite3.OperationalError, PermissionError):
+        printd(f"Failed to create {dbFile}, aborting")
+        raise
+
+    # create new table if it doesn't exist
+    try:
+      self.db.execute("select * from dirnotes")
+    except sqlite3.OperationalError:
+      self.db.execute("create table dirnotes (name TEXT, date DATETIME, size INTEGER, comment TEXT, comment_date DATETIME, author TEXT)")
+      self.db.execute("create index dirnotes_i on dirnotes(name)") 
+      print_d(f"Table dirnotes created")
+      # at this point, if a shared database is required, somebody needs to set perms to 0o666
+  
+    self.writable = True
+    try:
+      self.db.execute("pragma user_verson=0")
+    except sqlite3.OperationalError:
+      self.writable = False
+
+DATE_FORMAT   = "%Y-%m-%d %H:%M:%S"
+class UiHelper:
+  @staticmethod
+  def epochToDb(epoch):
+    return time.strftime(DATE_FORMAT,time.localtime(epoch))
+  @staticmethod
+  def DbToEpoch(dbTime):
+    return time.mktime(time.strptime(dbTime,DATE_FORMAT))
+  @staticmethod
+  def getShortDate(longDate):
+    now = time.time()
+    diff = now - longDate
+    if diff > YEAR:
+      fmt = "%b %e  %Y"
+    else:
+      fmt = "%b %e %H:%M"
+    return time.strftime(fmt, time.localtime(longDate))
+  @staticmethod
+  def getShortSize(fo):
+    if fo.isDir():
+      return " <DIR> "
+    elif fo.isLink():
+      return " <LINK>"
+    size = fo.getSize()
+    log = int((math.log10(size+1)-2)/3)
+    s = " KMGTE"[log]
+    base = int(size/math.pow(10,log*3))
+    return f"{base}{s}".strip().rjust(7)
+
+
+## one for each file
+## and a special one for ".." parent directory
+class FileObj:
+  """  The FileObj knows about both kinds of comments. """
+  def __init__(self, fileName, db):
+    self.fileName = os.path.abspath(fileName)     # full path; dirs end WITHOUT a terminal /
+    self.stat = os.lstat(self.fileName)
+    self.displayName = os.path.split(fileName)[1] # base name; dirs end with a /
+    if self.isDir():
+      if not self.displayName.endswith('/'):
+        self.displayName += '/'
+    self.date = self.stat.st_mtime
+    self.size = self.stat.st_size 
+    self.db = db
+
+  def getName(self):
+    """ returns the absolute pathname """
+    return self.fileName
+  def getDisplayName(self):
+    """ returns just the basename of the file; dirs end in / """
+    return self.displayName
+
+  def getDbData(self):
+    """ returns (comment, author, comment_date) """
+    if not hasattr(self,'dbCommentAuthorDate'):
+      cad = self.db.execute("select comment, author, comment_date from dirnotes where name=? order by comment_date desc",(self.fileName,)).fetchone()
+      self.dbCommentAuthorDate = cad if cad else (None, None, None)
+    return self.dbCommentAuthorDate
+  def getDbComment(self):
+    return self.getDbData()[0]
+
+  def getXattrData(self):
+    """ returns (comment, author, comment_date) """
+    if not hasattr(self,'xattrCommentAuthorDate'):
+      c = a = d = None
+      try:
+        c = os.getxattr(self.fileName, xattr_comment, follow_symlinks=False).decode()
+        a = os.getxattr(self.fileName, xattr_author, follow_symlinks=False).decode()
+        d = os.getxattr(self.fileName, xattr_date, follow_symlinks=False).decode()
+      except:  # no xattr comment
+        pass
+      self.xattrCommentAuthorDate = c,a,d
+    return self.xattrCommentAuthorDate
+  def getXattrComment(self):
+    return self.getXattrData()[0]
+
+  def setDbComment(self,newComment):
+    # how are we going to hook this?
+    #if not self.db.writable:
+    #  errorBox("The database is readonly; you cannot add or edit comments")
+    #  return
+    s = os.lstat(self.fileName)
+    try:
+      print_d(f"setDbComment db {self.db}, file: {self.fileName}")
+      self.db.execute("insert into dirnotes (name,date,size,comment,comment_date,author) values (?,datetime(?,'unixepoch','localtime'),?,?,datetime(?,'unixepoch','localtime'),?)",
+          (self.fileName, s.st_mtime, s.st_size,
+          str(newComment), time.time(), getpass.getuser()))
+      self.db.commit()
+      self.dbCommentAuthorDate = newComment, getpass.getuser(), UiHelper.epochToDb(time.time())
+    except sqlite3.OperationalError:
+      print_d("database is locked or unwritable")
+      errorBox("the database that stores comments is locked or unwritable")
+
+  def setXattrComment(self,newComment):
+    print_d(f"set comment {newComment} on file {self.fileName}")
+    try:
+      os.setxattr(self.fileName,xattr_comment,bytes(newComment,'utf8'),follow_symlinks=False)
+      os.setxattr(self.fileName,xattr_author,bytes(getpass.getuser(),'utf8'),follow_symlinks=False)
+      os.setxattr(self.fileName,xattr_date,bytes(time.strftime(DATE_FORMAT),'utf8'),follow_symlinks=False)
+      self.xattrCommentAuthorDate = newComment, getpass.getuser(), time.strftime(DATE_FORMAT) 
+      return True
+    # we need to move these cases out to a handler 
+    except Exception as e:
+      if self.isLink():
+        errorBox("Linux does not allow xattr comments on symlinks; comment is stored in database")
+      elif self.isSock():
+        errorBox("Linux does not allow comments on sockets; comment is stored in database")
+      elif os.access(self.fileName, os.W_OK)!=True:
+        errorBox(f"you don't appear to have write permissions on this file: {self.fileName}")
+        # change the listbox background to yellow
+      elif "Errno 95" in str(e):
+        errorBox("is this a VFAT or EXFAT volume? these don't allow comments")
+      return False
+
+  def getComment(self,mode):
+    """ returns the comment for the given mode """
+    return self.getDbComment() if mode == "db"    else self.getXattrComment()
+  def getOtherComment(self,mode):
+    return self.getDbComment() if mode == "xattr" else self.getXattrComment()
+  def getData(self,mode):
+    """ returns (comment, author, comment_date) for the given mode """
+    return self.getDbData()    if mode == "db"    else self.getXattrData()
+  def getOtherData(self,mode):
+    """ returns (comment, author, comment_date) for the 'other' mode """
+    return self.getDbData()    if mode == "xattr" else self.getXattrData()
+
+  def getDate(self):
+    return self.date
+  def getSize(self):
+    return self.size
+  def isDir(self):
+    return stat.S_ISDIR(self.stat.st_mode)
+  def isLink(self):
+    return stat.S_ISLNK(self.stat.st_mode)
+  def isSock(self):
+    return stat.S_ISSOCK(self.stat.st_mode)
+
+  def copyFile(self, dest, doMove = False):
+    """ dest is either a FQ filename or a FQ directory, to be expanded with same basename """
+    # NOTE: this method copies the xattr (comment + old author + old date) 
+    #       but creates new db (comment + this author + new date)
+    if os.path.isdir(dest):
+      dest = os.path.join(destDir,self.displayName)
+    try:
+      print_d("try copy from",self.fileName,"to",dest)
+      # shutil methods preserve dates & chmod/chown & xattr
+      if doMove:
+        shutil.move(self.fileName, dest)
+      else:
+        shutil.copy2(self.fileName, dest)  
+      # can raise FileNotFoundError, Permission Error, shutil.SameFileError, IsADirectoryError 
+    except:
+      errorBox(f"file copy/move to <{dest}> failed; check permissions")
+      return
+    # and copy the database record
+    f = FileObj(dest, self.db)
+    f.setDbComment(self.getDbComment())
+  def moveFile(self, dest):
+    """ dest is either a FQ filename or a FQ directory, to be expanded with same basename """
+    self.copyFile(dest, doMove = True)
+
+# >>> snip here <<<
 
 #============= the functions that are called from the main.loop ===============
 
-def file_copy(f,target,target_is_dir,force):
-    print_d(f"call file_copy with args={target},{target_is_dir} and {force}")
-    dest = target if not target_is_dir else target+'/'+os.path.basename(f)
+def file_copy(f,target,target_is_dir,is_copy,force):
+    print_d(f"call file_copy/move with args={target},{target_is_dir} and {force}")
+    dest = target if not target_is_dir else os.path.join(target,f.getDisplayName())
     if os.path.exists(dest) and not force:
-        go = input("The copy target <<" + dest + ">> exists. Overwrite? (y or n) ")
+        go = input("The copy/move target <<" + dest + ">> exists. Overwrite? (y or n) ")
         if go != 'y' and go != 'Y':
             return
-    print_d(f"copy from {f} to {dest}")
+    print_d(f"copy/move from {f} to {dest}")
+    if is_copy:
+      f.copyFile(dest)
+    else:
+      f.moveFile(dest)
 
 def file_zap(f,all_flag):
-    print_d(f"zapping the comment history of {f}")
+    db = f.db
+    print_d(f"zapping the comment history of {f.getName()}")
     if all_flag:
-        print_d("zapping the entire database")
+        confirm = input("You requested a complete flush of the comment database history. Please hit 'Y' to confirm")
+        if confirm == 'Y':
+            print_d("zapping the entire database")
+            db.execute("delete from dirnotes where comment_date < (select max(comment_date) from dirnotes d2 where d2.name = dirnotes.name)")
+    else:
+        db.execute("delete from dirnotes where name=? and comment_date < (select max(comment_date) from dirnotes where name=?)",(f.getName(),f.getName()))
+    db.commit()
 
 def file_modify_comment(f, create, append, erase):
     print_d(f"modify the comment on file {f} with extra={(create,append,erase)}")
-    if not os.path.exists(f):
+    if not os.path.exists(f.getName()):
         print(f"the target file does not exist; please check the spelling of the file: {f}")
-        # sys.exit() here?
+        sys.exit(10)
+    if create:
+        f.setXattrComment(create)
+        f.setDbComment(create)
+    elif append:
+        c = f.getComment(mode)
+        f.setXattrComment(f"{c}; {append}")
+        f.setDbComment(f"{c}; {append}")
+    elif erase:
+        f.setXattrComment('')
+        f.setDbComment('')
 
-def file_display(f, listall, history, json, minimal):
-    print_d(f"list file details {f}")
-    x_comment = None
-    try:
-        x_comment = xattr.getxattr(f,xattr_comment).decode()
-        x_author    = xattr.getxattr(f,xattr_author).decode()
-        x_date      = xattr.getxattr(f,xattr_date).decode()
-    except:
-        pass
-    full_f = os.path.realpath(f)
-
-    d_comment = getDbComment(full_f)
-    if d_comment:
-        d_comment, d_author, d_date = d_comment
-    print_d(f"for file {f}, database comment is <{d_comment}>, xattr comment is <{x_comment}>")
-
-    if os.path.isdir(f):
-        f = f+'/'
-    if x_comment or listall:
-        if x_comment and (d_comment != x_comment):
-            x_comment += '*'
+def file_display(f, listall, json, minimal):
+    fn = f.getDisplayName()
+    print_d(f"list file details {fn}")
+    c,a,d = f.getData(mode)
+    c1,a1,d1 = f.getOtherData(mode)
+
+    if c or listall:
+        if c and (c != c1):
+            c += '*'
         if not json:
             if minimal:
-                print(f"{x_comment}")
+                print(f"{c}")
+            elif verbose:
+                print(f"{f.getName()}: {repr(c)}, {repr(a)}, {repr(d)}")
             else:
-                print(f"{f}: {x_comment}")
+                print(f"{fn}: {repr(c)}")
         else:
             if verbose:
-                answer_json.append( {"file":f,"comment":x_comment,"author":x_author,"date":x_date } )
+                answer_json.append( {"file":f.getName(),"comment":c,"author":a,"date":d } )
             else:
-                answer_json.append( {"file":f,"comment":x_comment} )
-
-def getDbComment(full_filename):
-    global db
-    print_d(f"db access for {full_filename}")
-    c = db.cursor()
-    c.execute("select comment,author,comment_date from dirnotes where name=? and comment<>'' order by comment_date desc",(full_filename,))
-    a = c.fetchone()
-    if a:
-        return a[0:3]
-
-def openDb():
-    global db
-    dbName = "~/.dirnotes.db"
-    dbName = os.path.expanduser(dbName)
-    db = sqlite3.connect(dbName)
-    try:
-        c = db.cursor()
-        c.execute("select * from dirnotes")
-    except sqlite3.OperationalError:
-        c.execute("create table dirnotes (name TEXT, date DATETIME, size INTEGER, comment TEXT, comment_date DATETIME, author TEXT)")
-    return db 
+                answer_json.append( {"file":fn,"comment":c} )
+
+def file_history(f,json):
+    db = f.db
+    c = db.execute("select comment, author, comment_date from dirnotes where name=? order by comment_date desc",(f.getName(),))
+    if not json:
+        print(f"file: \t\t{f.getName()}\n")
+    else:
+        answer_json.append ( {"file":f.getName()} )
+    for a in c.fetchall():
+        if not json:
+            print(f"comment: \t{a[0]}\nauthor: \t{a[1]}\t\tdate: \t\t{a[2]}\n")
+        else:
+            answer_json.append( {"comment":a[0],"author":a[1],"date":a[2]} )
+
 
 def main(args):
     parser = argparse.ArgumentParser(description="Display or add comments to files",
         epilog="Some options conflict. Use only one of: -l -c -a -H -e -z -Z and one of -d -x")
-    parser.add_argument('-V',"--version", action="version",     version=f"dncli ver:{VERSION}")
-    parser.add_argument('-v',"--verbose", action='count',           help="verbose, almost debugging; do not use in scripts",default=0)
-    parser.add_argument('-j',"--json",      action="store_true",help="output in JSON format")
+    parser.add_argument('-V',"--version", action="version",   version=f"dncli ver:{VERSION}")
+    parser.add_argument('-v',"--verbose", action='count',     help="verbose output (include comment author & date)",default=0)
+    parser.add_argument('-D',"--debug",   action='store_true',help="include debugging output; do not use in scripts",default=0)
+    parser.add_argument('-j',"--json",    action="store_true",help="output in JSON format")
     pars_m = parser.add_mutually_exclusive_group()
     pars_m.add_argument('-l',"--listall", action="store_true",help="list all files, including those without comments")
-    parser.add_argument('-d',"--db",            action="store_true",help="list comments from database")
-    parser.add_argument('-x',"--xattr",     action="store_true",help="list comments from xattr")
+    parser.add_argument('-d',"--db",      action="store_true",help="list comments from database")
+    parser.add_argument('-x',"--xattr",   action="store_true",help="list comments from xattr")
     parser.add_argument('-n',"--minimal", action="store_true",help="output only comments; useful in scripting")
-    parser.add_argument('-H',"--history", action="store_true",help="output the history of comments for a file")
-    pars_m.add_argument('-c',"--create",    metavar="comment",  help="add a comment to a file")
-    pars_m.add_argument('-a',"--append",    metavar="comment",  help="append to a comment on a file, separator=';'")
-    pars_m.add_argument('-C',"--copy",      action="store_true",help="copy a file with its comments")
+    parser.add_argument('-H',"--history", action="store_true",help="output the history of database comments for a file")
+    pars_m.add_argument('-c',"--create",  metavar="comment",  help="add a comment to a file")
+    pars_m.add_argument('-a',"--append",  metavar="comment",  help="append to a comment on a file, separator=';'")
+    pars_m.add_argument('-C',"--copy",    action="store_true",help="copy a file with its comments")
+    pars_m.add_argument('-M',"--move",    action="store_true",help="move a file with its comments")
     parser.add_argument('-y',"--cp_force",action="store_true",help="copy over existing files")
-    pars_m.add_argument('-e',"--erase",     action="store_true",help="erase the comment on a file")
-    pars_m.add_argument('-z',"--zap",           action="store_true",help="clear the comment history on a file")
-    pars_m.add_argument('-Z',"--zapall",     action="store_true",help="clear the comment history in the entire database")
+    pars_m.add_argument('-e',"--erase",   action="store_true",help="erase the comment on a file")
+    pars_m.add_argument('-z',"--zap",     action="store_true",help="clear the database comment history on a file")
+    pars_m.add_argument('-Z',"--zapall",  action="store_true",help="clear the comment history in the entire database")
+    parser.add_argument(     "--config",  dest="config_file", help="use config file (default ~/.config/dirnotes/dirnotes.conf)")
     parser.add_argument('file_list',nargs='*',help="file(s); list commands may omit this")
     args = parser.parse_args()
 
@@ -140,9 +392,16 @@ def main(args):
         print("the -x/--xattr option doesn't apply to the -z/--zap and -Z/--zapall commands")
         sys.exit(7)
 
-    global verbose
+    global verbose, debug
     verbose = args.verbose
+    debug = args.debug
     
+    config = ConfigLoader(args.config_file or DEFAULT_CONFIG_FILE)
+    global mode
+    mode = config.mode
+    mode = "xattr" if args.xattr else ("db" if args.db else mode)
+    db = DnDataBase(config.dbName).db
+
     #====== 1) build the file list =============
 
     files = args.file_list
@@ -150,7 +409,7 @@ def main(args):
     if not files and args.display:
         files = os.listdir(".")
         files.sort()
-    # other command require explicity file lists, but 'dncli -c "new comment" *' will work
+    # other command require explicity file lists
     if not files:
         print("please specify a file or files to use")
         sys.exit(10)
@@ -166,36 +425,36 @@ def main(args):
         func = file_zap
         loop_args = (args.zapall,)
         if args.zapall:
-            files = (1,)
-    elif args.copy:
-        print_d(f"got a copy command to {args.copy}")
+            files = ('.',)
+    elif args.copy or args.move:
+        print_d(f"got a copy/move command to copy={args.copy}, move={args.move}")
         # the last item on the file list is the target
         n_files = len(files)
         if n_files < 2:
-            print_d(f"the copy command requires at least two arguments, the last one is the destination")
+            print("the copy/move command requires at least two arguments, the last one is the destination")
             sys.exit(1)
-        target = files[-1]
+        files, target = files[:-1], files[-1]
         target_is_directory = os.path.isdir(target)
-        files = files[:-1]
-        print_d(f"copy from {files} to {target}")
+        print_d(f"copy/move from {files} to {target}")
         if n_files > 2 and not target_is_directory:
-            print_d(f"multiple copy files must go to a target directory")
+            print("multiple copy/move files must go to a target directory")
             sys.exit(3)
         func = file_copy
-        loop_args = (target, target_is_directory, args.cp_force)
+        loop_args = (target, target_is_directory, args.copy, args.cp_force)
+    elif args.history:
+        func = file_history
+        loop_args = (args.json,)
     else:
-        assert(args.display)
-        print_v(f"list files using {'database' if args.db else 'xattr'} priority")
+        assert args.display
+        print_d(f"list files using {mode} priority")
         print_d(f"display command with option: {args.listall} and {args.history} and {args.json} and {args.minimal}")
-        loop_args = (args.listall, args.history, args.json, args.minimal)
+        loop_args = (args.listall, args.json, args.minimal)
         func = file_display
 
     #====== 3) loop on the list, execute the function =============
-    global db
-    db = openDb()
 
     for f in files:
-        func(f,*loop_args)
+        func(FileObj(f,db),*loop_args)
 
     if answer_json:
         print(json.dumps(answer_json))

dirnotes-cli.md

@@ -26,8 +26,8 @@ dirnotes-cli - view and manage Dirnotes file annotations
     **dirnotes-cli** -e filename...
 
   FileCopy:  
-    **dirnotes-cli** -C src-file... <dest-file | dest-dir>  
-    **dirnotes-cli** -M src-file... <dest-file | dest-dir>
+    **dirnotes-cli** -C [-y] src-file... <dest-file | dest-dir>  
+    **dirnotes-cli** -M [-y] src-file... <dest-file | dest-dir>
 
   Cleanup:  
     **dirnotes-cli** -z [filename]...   
@@ -41,31 +41,39 @@ a file. The descriptions are stored in two places:
   * in the xattr properties of the file
   * in a database located in the user's home directory
 
+The **dirnotes-cli** is designed for use on the command line, or in
+scripting.
 
 The *list* commands will display the comment from the database or xattr (as
-determined by the config file, below, or as specified with **-d**/**-x** options). If the database and xattr comments differ, the
-comment will be terminated by a '\*' character. The _-H_ option displays the history 
-of comments from the database.
+determined by the config file or as specified with **-d**/**-x** options). 
+If the database and xattr comments differ, the
+comment will be terminated by a '\*' character. 
+The **-H** option displays the history of comments from the database.
 
-The output of the *list* commands can be in .json format (_-j_) , and can optionally 
-display the comment creator and the comment date (_-v_)
+The output of the *list* commands can be in .json format (**-j**) , 
+and can optionally display the comment creator and the comment date (**-v**)
 
-The *create* commands will attempt to store the file comments in both the xattr of the file,
-and in the database. If either of these fail, they fail silently. Use the _-c_ to create a comment, use _-a_ to append to an existing comment, and _-e_ to erase a comment.
+The *create* commands will attempt to store the file comments in _both_ 
+the xattr of the file, and in the database. 
+If either of these fail, they fail silently. Use the **-c** to create a comment, 
+use **-a** to append to an existing comment, and **-e** to erase a comment.
 
-The *filecopy* commands will copy the file to a destination, and preserve the file
-comments. [See notes below about LIMITATIONS]
+The *filecopy* commands will copy/move the file to a destination, and 
+preserve the file comments. [See notes below about LIMITATIONS]
 
 The *cleanup* commands can clean up the history of comments in the database.
 
 # LIMITATIONS
 
-The file comments are located in two locations: a database, and in the xattr properties
-of the file. Each of these storage locations has its own benefits and limitations:
+The file comments are located in two locations: a database, and in the xattr 
+properties of the file. Each of these storage locations has its own benefits 
+and limitations:
 
 ## xattr
 
-Comments stored in the xattr properties can be copied/moved with the file, if you
+Because _xattr_ comments are bound to the filesystem, other command line tools 
+may be used to manage them. Comments stored in the _xattr_ properties can be 
+copied/moved with the file, if you
 use the correct options for _cp_. The _mv_ utility automatically preserves _xattr_.
 Other programs can also be coerced into perserving _xattr_ properties:
 
@@ -73,53 +81,64 @@ Other programs can also be coerced into perserving _xattr_ properties:
 * tar
 * mksquashfs
 
-Not all file systems support xattr properties (vfat/exfat does not). 
+Not all file systems support _xattr_ properties (vfat/exfat does not). 
 
-The current implementation of _sshfs_ and _scp_ do not support the copy of _xattr_ properties. 
-If you want to copy files to a remote machine and include the _xattr_ comments, use _rsync_ with the _-X_ option. Or _tar_ of course.
+The current implementation of _sshfs_ and _scp_ do not support the copy of 
+_xattr_ properties. If you want to copy files to a remote machine and 
+include the _xattr_ comments, use _rsync_ with the _-X_ option. Or _tar_ of course.
 
-Some editing apps (like _vim_) will create a new file when saving the data, which orphans the _xattr_ comments. For these apps, use the _database_ system.
+Some editing apps (like _vim_) will create a new file when saving the data, 
+which orphans the _xattr_ comments. For these apps, use the _database_ system.
+
+Of course, once you start to manipulate _xattr_ comments outside of the **dirnotes**
+programs, the _xattr_ comments will become out of sync with the _database_ comments.
 
 
 ## database
 
-Comments stored in the database work for all filesystem types (including vfat/exfat/sshfs)
-Comments are personalized to the _current user_. 
-Another user on the same system will not see these comments.
+Comments stored in the database work for all filesystem types 
+(including vfat/exfat/sshfs)
+Comments are usually stored in a _per user_ database; thus another user on 
+the same system will not see these comments.
 
 Files are indexed by their complete path name. Removable filesystems should be
 mounted in a consistent way, so that the complete path name is reproducable.
 
+If you are using _sshfs_, you can use the **dirnotes** programs to copy a file, 
+and the database comments will work properly.
+
 Comments stored in the database do not travel with the files when
- they are moved or copied, unless using the dirnotes-family of tools.
+they are moved or copied outside of using the **dirnotes**-family of tools.
 
-If you are using _sshfs_, you can use the 'dirnotes' programs to copy a file, and the database comments will work properly.
 
 # OPTIONS
   
+**-a**
+: append to a comment on a file
+
 **-c**
 : add a comment to a file
 
 **-C**
-: attempt to copy the file(s) and comments to a destination; if multiple files are copied, the destination must be a directory
+: attempt to copy the file(s) and comments to a destination; if multiple files are copied, the destination must be a directory; the last argument is the destination
 
 **-d**
 : use database comments as the primary source; cannot be used with **-x**
 
+**-D**
+: print debugging information
+
 **-e**
-: erase the comments on a file
+: erase the comments on the file(s)
 
 **-h** **--help**
 : display help and usage
 
 **-H**
-: output the history of comments for this file(s)
-
-**-i**  
-: add a comment to a file; if the comment is not in the command, accept it from stdin
+: output the history of comments for the file(s)
 
 **-j**
-: output (to stdio) the file comment in .json format 
+: output (to stdout) the file comment in .json format 
 
 **-l**
 : list all files, including those without _dirnotes_ comments
@@ -131,7 +150,7 @@ If you are using _sshfs_, you can use the 'dirnotes' programs to copy a file, an
 : output only the comment; this may be useful for scripting
 
 **-v**
-: also include the comment author and comment date in the output
+: list full path names, also include the comment author and date in the output
 
 **-V** **--version**
 : display version number
@@ -139,8 +158,11 @@ If you are using _sshfs_, you can use the 'dirnotes' programs to copy a file, an
 **-x**
 : use xattr comments as the primary source; connot be used with **-d**
 
+**-y**
+: allow file overwrite for **-C** copy or **-M** move
+
 **-z**
-: erase history comments associated with this file; keep the current comment; if no file is specified, erase the history comments for all files in the current directory
+: erase history comments associated with the file(s); keep the current comment
 
 **-Z**
 : erase all the historic comments in the user's database
@@ -149,19 +171,24 @@ If you are using _sshfs_, you can use the 'dirnotes' programs to copy a file, an
 
 To display the comment for a file:
 
-> $ dirnotes-cli filename.txt  
-> filename.txt: notes on the car repair
+> <code>$ dirnotes-cli car_notes.txt</code>  
+> <code>car_notes.txt: 'notes on the car repair'</code>
 
 To extract _only_ the comment from a file, use the _-n_ option (useful for scripts):
 
-> $ dirnotes-cli -n filename.txt  
-> notes on the car repair
+> <code>$ dirnotes-cli -n car_notes.txt</code>    
+> <code>notes on the car repair</code>
 
 Or, in json format:
 
-> $ dirnotes-cli -j filename.txt  
-> [{"file": "filename.txt", "comment": "notes on the car repair"}]
+> <code>$ dirnotes-cli -j car_notes.txt</code>  
+> <code>[{"file": "car_notes.txt", "comment": "notes on the car repair"}]</code>
+
+To append to a comment:
 
+> <code>$ dirnotes-cli -a 'first quote: \$1,400' car_notes.txt</code>  
+> <code>$ dirnotes-cli car_notes.txt</code>  
+> <code>car_notes.txt: 'notes on the car repair; first quote: $1,400'</code>
 
 
 # SEE ALSO
@@ -179,9 +206,10 @@ The **dirnotes-cli** program provides command line access, and can be scripted.
 By default, the file **~/.dirnotes.conf** will be used to load the user's config. There are
 three attributes described in that file that are important:
 
-> * xattr_tag  
-> * database  
-> * start_mode
+> * xattr_tag  [default: user.xdg.comment]   
+> * database   [default: ~/.local/share/dirnotes/dirnotes.db]  
+> * start_mode [default: xattr]  
 
+The default location for the config file is ~/.config/dirnotes/dirnotes.conf
 
 

dirnotes-tui

@@ -4,6 +4,8 @@
 # TODO: re-read date/author to xattr after an edit
 # TODO: consider adding h,j,k,l movement
 # TODO: change move command to 'v', change mode to 'm', drop copy-comments
+# TODO: bug: enter db mode, type E to edit a comment, we get the xattr version!!
+# TODO: try to clear a comment, left with ' '
 
 # scroll
 # up/down - change focus, at limit: move 1 line,
@@ -22,16 +24,13 @@ import curses, sqlite3, curses.textpad
 import logging, getpass, argparse
 import json
 
-VERSION = "1.8"
+VERSION = "1.9"
 # these may be different on MacOS
 xattr_comment = "user.xdg.comment"
 xattr_author  = "user.xdg.comment.author"
 xattr_date    = "user.xdg.comment.date"
 DATE_FORMAT   = "%Y-%m-%d %H:%M:%S"
 
-# convert the ~/ form to a fully qualified path
-# database_name = "~/.dirnotes.db"
-
 mode_names = {"db":"<Database mode> ","xattr":"<Xattr mode>"}
 modes = ("db","xattr")
 mode = "db"
@@ -191,8 +190,8 @@ class Pane:
     self.file_pad.addstr(y,self.sep1,UiHelper.getShortSize(f))
     self.file_pad.addstr(y,self.sep2,UiHelper.getShortDate(f.date))
 
-    comment = f.getComment() or ''
-    other   = f.getOtherComment() or ''
+    comment = f.getComment(mode) or ''
+    other   = f.getOtherComment(mode) or ''
     logging.info(f"file_line, comments are <{comment}> and <{other}>  differ_flag:{self.some_comments_differ}")
     if comment == other:
       self.file_pad.addnstr(y,self.sep3,comment,self.w-self.sep3-2)
@@ -286,20 +285,22 @@ class Files():
 
   def sortSize(a):
     if a.getDisplayName() == '../':
-      return 0
+      return -2
+    if a.isDir() or a.isLink() or a.isSock():
+      return -1
     return a.getSize()
 
   def sortComment(a):
-    return a.getComment()
+    return a.getComment(mode) or '~'
 
-  sort_mode = sortName
+  sortFunc = sortName
   def sort(self):
-    self.files.sort(key = Files.sort_mode)
+    self.files.sort(key = Files.sortFunc)
 
   def getCurDir(self):
     return self.directory
   def getMasterComment(self):
-    return self.directory.getComment()
+    return self.directory.getComment(mode)
 
   ## accessors ##
   def __len__(self):
@@ -409,7 +410,7 @@ class UiHelper:
     now = time.time()
     diff = now - longDate
     if diff > YEAR:
-      fmt = "%b %e %Y"
+      fmt = "%b %e  %Y"
     else:
       fmt = "%b %e %H:%M"
     return time.strftime(fmt, time.localtime(longDate))
@@ -438,16 +439,18 @@ class FileObj:
       if not self.displayName.endswith('/'):
         self.displayName += '/'
     self.date = self.stat.st_mtime
-    self.size = self.stat.st_size
+    self.size = self.stat.st_size 
     self.db = db
 
   def getName(self):
+    """ returns the absolute pathname """
     return self.fileName
   def getDisplayName(self):
+    """ returns just this basename of the file; dirs end in / """
     return self.displayName
 
-  # new methods; no caching
   def getDbData(self):
+    """ returns (comment, author, comment_date) """
     if not hasattr(self,'dbCommentAuthorDate'):
       cad = self.db.execute("select comment, author, comment_date from dirnotes where name=? order by comment_date desc",(self.fileName,)).fetchone()
       self.dbCommentAuthorDate = cad if cad else (None, None, None)
@@ -456,6 +459,7 @@ class FileObj:
     return self.getDbData()[0]
 
   def getXattrData(self):
+    """ returns (comment, author, comment_date) """
     if not hasattr(self,'xattrCommentAuthorDate'):
       c = a = d = None
       try:
@@ -507,13 +511,15 @@ class FileObj:
         errorBox("is this a VFAT or EXFAT volume? these don't allow comments")
       return False
 
-  def getComment(self):
+  def getComment(self,mode):
     return self.getDbComment() if mode == "db"    else self.getXattrComment()
-  def getOtherComment(self):
+  def getOtherComment(self,mode):
     return self.getDbComment() if mode == "xattr" else self.getXattrComment()
-  def getData(self):
+  def getData(self,mode):
+    """ returns (comment, author, comment_date) """
     return self.getDbData()    if mode == "db"    else self.getXattrData()
-  def getOtherData(self):
+  def getOtherData(self,mode):
+    """ returns (comment, author, comment_date) """
     return self.getDbData()    if mode == "xattr" else self.getXattrData()
 
   def getDate(self):
@@ -533,18 +539,33 @@ class FileObj:
     if stat.S_ISREG(self.stat.st_mode):
       dest = os.path.join(destDir,self.displayName)
       try:
-        print("try copy from",self.fileName,"to",dest)
+        print_v("try copy from",self.fileName,"to",dest)
         shutil.copy2(self.fileName, dest)
       except:
         errorBox(f"file copy to <{dest}> failed; check permissions")
+        return
       f = FileObj(dest, self.db)
-      print("dest object created",repr(f))
       f.setDbComment(self.getDbComment())
-
-##########  dest folder picker ###############
+  def moveFile(self, destDir):
+    # NOTE: this method moves the xattr (comment + old author + old date)
+    #       but creates new db (comment + this author + new date)
+    src  = self.fileName
+    dest = os.path.join(destDir, self.displayName)
+    # move preserves dates & chmod/chown & xattr
+    print_v(f"move from {self.fileName} to {destDir}")
+    try:
+      shutil.move(src, dest)
+    except:
+      ErrorBox(f"file move to <{dest}> failed; check permissions")
+      return
+    # and copy the database record
+    f = FileObj(dest,files.db)
+    f.setDbComment(self.getDbComment())  
+    
+##########  dest directory picker ###############
 # returns None if the user hits <esc>
 #     the dir_pad contents are indexed from 0,0, matching self.fs
-class showFolderPicker:
+class showDirectoryPicker:
   def __init__(self,starting_dir,title):
     self.selected = None
     self.title = title
@@ -744,11 +765,11 @@ Comments detail:
 def show_detail(f):
   global mode
   h = paint_dialog(COLOR_HELP,detail_string)
-  c,a,d = f.getData()   # get all three, depending on the current mode
+  c,a,d = f.getData(mode)   # get all three, depending on the current mode
   h.addstr(1,20,"from xattrs" if mode=="xattr" else "from database")
-  h.addnstr(2,12,c,h.getmaxyx()[1]-13)
-  h.addstr(3,12,a if a else "<not set>")
-  h.addstr(4,12,d if d else "<not set>")
+  h.addnstr(2,12,c or "<not set>",h.getmaxyx()[1]-13)
+  h.addstr(3,12,a or "<not set>")
+  h.addstr(4,12,d or "<not set>")
   h.refresh()
   c = h.getch()
   del h
@@ -807,13 +828,13 @@ def main(w, cwd, database_file, start_file):
     elif c == CMD_SORT:
       c = show_sort()
       if c == ord('s') or c == ord('S'):
-        Files.sort_mode = Files.sortSize
+        Files.sortFunc = Files.sortSize
       elif c == ord('n') or c == ord('N'):
-        Files.sort_mode = Files.sortName
+        Files.sortFunc = Files.sortName
       elif c == ord('d') or c == ord('D'):
-        Files.sort_mode = Files.sortDate
+        Files.sortFunc = Files.sortDate
       elif c == ord('c') or c == ord('C'):
-        Files.sort_mode = Files.sortComment
+        Files.sortFunc = Files.sortComment
       files.sort()
       mywin.refill()
       mywin.refresh()
@@ -849,7 +870,7 @@ def main(w, cwd, database_file, start_file):
       f = files[mywin.cursor]
       if f.isDir():
         cwd = f.getName()
-        logging.info(f"CD change to {cwd}")
+        print_v(f"CD change to {cwd}")
         files = Files(cwd,db)
         mywin = Pane(w,cwd,files)
         # TODO: should this simply re-fill() the existing Pane instead of destroy?
@@ -915,43 +936,29 @@ def main(w, cwd, database_file, start_file):
       mywin.refresh()
 
     elif c == CMD_COPY:
-      if files[mywin.cursor].displayName == "..":
-        continue
-      if os.path.isdir(files[mywin.cursor].fileName):
-        errorBox(f"<{files[mywin.cursor].displayName}> is a directory. Copy not allowed")
+      if files[mywin.cursor].getDisplayName() == "../":
         continue
-      dest_dir = showFolderPicker(cwd,"Select folder for copy").value()
-      if dest_dir:
-        #errorBox(f"copy cmd to {dest_dir}")
-        src = cwd + '/' + files[mywin.cursor].displayName
-        dest = dest_dir + '/' + files[mywin.cursor].displayName
-        # copy2 preserves dates & chmod/chown & xattr
-        logging.info(f"copy from {src} to {dest_dir}")
-        shutil.copy2(src, dest_dir)
-        # and copy the database record
-        f = FileObj(dest,files.db) 
-        f.setDbComment(files[mywin.cursor].getDbComment())
+      if files[mywin.cursor].isDir():
+        errorBox(f"<{files[mywin.cursor].getDisplayName()}> is a directory. Copy not allowed")
+      else:
+        dest_dir = showDirectoryPicker(cwd,"Select folder for copy").value()
+        if dest_dir:
+          files[mywin.cursor].copyFile(dest_dir)
       mywin.refresh() 
 
     elif c == CMD_MOVE:
-      if files[mywin.cursor].displayName == "..":
+      if files[mywin.cursor].getDisplayName() == "../":
         continue
-      if os.path.isdir(files[mywin.cursor].fileName):
-        errorBox(f"<{files[mywin.cursor].displayName}> is a directory. Move not allowed")
-        continue
-      dest_dir = showFolderPicker(cwd,"Select folder for move").value()
-      if dest_dir:
-        #errorBox(f"move cmd to {dest_dir}")      
-        src = cwd + '/' + files[mywin.cursor].displayName
-        dest = dest_dir + '/' + files[mywin.cursor].displayName
-        # move preserves dates & chmod/chown & xattr
-        logging.info(f"move from {src} to {dest_dir}")
-        shutil.move(src, dest_dir)
-        # and copy the database record
-        f = FileObj(dest,files.db)
-        f.setDbComment(files[mywin.cursor].getDbComment())  
-        files = Files(cwd,db)
-        mywin = Pane(w,cwd,files) # is this the way to refresh the main pane? TODO
+      if files[mywin.cursor].isDir():
+        errorBox(f"<{files[mywin.cursor].getDisplayName()}> is a directory. Move not allowed")
+      else:
+        dest_dir = showDirectoryPicker(cwd,"Select folder for move").value()
+        if dest_dir:
+          files[mywin.cursor].moveFile(dest_dir)
+          files = Files(cwd,db)
+          mywin = Pane(w,cwd,files) # is this the way to refresh the main pane? TODO
+      mywin.refresh()   # to clean up the errorBox or FolderPicker
+        
 
     elif c == curses.KEY_RESIZE:
       mywin.resize()
@@ -965,12 +972,17 @@ def pre_main():
   parser = argparse.ArgumentParser(description="Add comments to files")
   parser.add_argument('-c','--config',  dest='config_file', help="config file (json format)")
   parser.add_argument('-v','--version', action='version', version=f"dirnotes ver:{VERSION}")
+  parser.add_argument('-d','--db', action='store_true',help="start up in database mode")
+  parser.add_argument('-x','--xattr', action='store_true',help="start up in xattr mode")
   parser.add_argument('directory', type=str, default='.', nargs='?',  help="directory or file to start")
   args = parser.parse_args()
   logging.info(args)
 
   config = ConfigLoader(args.config_file or DEFAULT_CONFIG_FILE)
-
+  if args.db:
+    config.mode = "db"
+  if args.xattr:
+    config.mode = "xattr"
   # print(repr(config))
   # print("start_mode",config["start_mode"])
   
@@ -992,7 +1004,4 @@ else:
 curses_running = True
 curses.wrapper(main, cwd or '.', database_name, start_file)
 
-# dirnotes database is name, date, size, comment, comment_date, author
 
-# symlinks: follow_symlinks should always be True, because symlinks in Linux
-#    can't have xattr....it appears to be the same in darwin