Wednesday, January 18, 2017

Issues / Problems

This post is just a place for issues / problems /discussion, should they arise.

154 comments:

  1. Anyone else having issues downloading from Discover? Just got an error today saying "You don't have permission to access "http:\\ofx.discovercard.com\" on this server. Not sure if this is a temporary glitch, or a change in protocol. Will watch for a bit before acting...

    ReplyDelete
  2. I've been having problems with Discover all this week.... I am getting an HTTP error 403, which is a "forbidden" error, and not being able to connect...

    Harold

    ReplyDelete
  3. I posted the same comment on my blog too. I haven't had any success by calling their customer service web support. It is difficult to get routed to the right tech support person that can resolve the server issue. I tried claiming I use Quicken, and they say it's their problem. Then I tried saying I use Microsoft Money and they say "we don't support that program". Looking at a QFX file from web downloads doesn't offer much help but it does list the FID as 9625 instead of 7101.

    ReplyDelete
  4. From the Moneydance forum at
    http://help.infinitekind.com/discussions/online-banking/4953-discover-downloads-stopped-working

    Quoting:
    Posted by robf on Feb 24, 2017 @ 11:03 AM

    robf's Avatar
    I HAVE THE SOLUTION!

    The HTTP request User Agent header has changed. In my code (I'm not using Moneydance but my own creation), the Discover server used to accept this header: httpReqst.UserAgent = "httpclient"

    Now, if I comment out that header or set httpReqst.UserAgent =null I can download from Discover; if it's left uncommented I get the 403 Forbidden error

    The Moneydance software guys (and gals) should update their OFX request with this change for Discover. Be warned, other institutions, such as TD Bank, require the UserAgent header to be set.

    ReplyDelete
    Replies
    1. Sounds promising. I'll look into it for the scripts... hopefully this weekend. Thanks Andrew!

      Delete
    2. Hi Robert,

      Thank you for your attention to this matter.
      Here are some direct links to the posts from robf on the Moneydance forums regarding his findings with Discover.

      http://help.infinitekind.com/discussions/online-banking/4953-discover-downloads-stopped-working#comment_42026919

      http://help.infinitekind.com/discussions/online-banking/4953-discover-downloads-stopped-working#comment_42027042

      http://help.infinitekind.com/discussions/online-banking/4953-discover-downloads-stopped-working#comment_42027064

      HTH -Kevin N.

      Delete
    3. Over at ofxhome, robler posted at
      http://www.ofxhome.com/ofxforum/viewtopic.php?id=47793

      Quoting:
      Here's the full HTTP header list that works for Discover:
      httpReqst.ContentLength = enc.Length;
      httpReqst.Method = "POST";
      httpReqst.ContentType = "application/x-ofx";
      httpReqst.UserAgent = null;
      httpReqst.ServicePoint.Expect100Continue = false;
      The first 3 are necessary for OFX requests at any institution. The last 2 are Discover specific, meaning other institutions might require other values.

      End Quoting.

      I don't know if robf and robler are the same person. According to MSDN, the default Expect100Continue value is true.

      Delete
  5. I am glad to see I am not the only one having issues - this stared about 7 - 10 days ago for me but my error is slightly different:

    ** An ERROR occurred sending POST request to ofx.discovercard.com
    Exception type:
    Exception Val : {Errno 10061} No connection could be made because the target machine actively refused it.

    As others have in this thread, I did reach out to Discover but they only provided assistance for manual download/import.

    On a side note, I cannot express how thankful I am to have this site and your help and input in keeping Microsoft Money alive for us Robert! Thank you so much!

    ReplyDelete
  6. Hi, i've absolutely had it with the crap called quicken 2011 & 2014 and determined not to buy 2017. I really don't trust it to add a column of numbers correctly (i've got evidence). Would anyone know a way to export data out of Quicken into Money keeping the categories & data? Got an aweful lot over these 6 years. thanks

    ReplyDelete
  7. You may have the most success by removing most of Money's categories in a blank data file and then setting up new categories in the same fashion as your Quicken has them prior to importing the QIF file created by Quicken. I'm not sure if you should setup the accounts prior to the import or if Money will do that for you.

    The expert on the subject is Dan (support at gaiersoftware.com) as he wrote the converter for Money to Quicken back in 2009.

    Let me know of your experience and I'll include it in my blog: https://microsoftmoneyoffline.wordpress.com/

    - ameridan

    ReplyDelete
  8. Just FYI... but my weekend turned out to be *busy*, so I haven't had time to dig into the Discover issue yet.

    ReplyDelete
  9. It seems BECU has gone through some system changes and now users are unable to connect.
    http://help.infinitekind.com/discussions/online-banking/5046-boeing-employees-credit-union-aka-becu-has-new-ofx-url has an answer from BECU, but doesn't seem to help.

    ReplyDelete
    Replies
    1. Tried various combinations and could not get it to work. I've messaged BECU again to try to get the correct url. They've changed this a couple of times in the past and it has been difficult to get the correct path. It appears they are using ofxver 102, so that should not be the issue.
      KR

      Delete
    2. Just fyi... but manual downloads seem to always be ofxver 102, but direct-connect may not be.

      Delete
  10. More info on it here. http://www.ofxhome.com/ofxforum/viewtopic.php?id=47793

    ReplyDelete
    Replies
    1. Yes... I actually sent a request to @robler, looking for the same info being requested there. I have a few ideas, but it will be more involved to test than I've had time to try... yet.

      Delete
    2. Got an answer from BECU, I haven't tried any changes yet:

      Thank you for contacting us. I’m sorry you’re having trouble connecting to BECU from your external money management program. I believe I can help.

      Following the last update to the BECU website last weekend we’ve received reports from other members that connections to their third-party accounting programs were interrupted during the update. To fix this, you should reset the connection. How this is done varies from one program to another, so if you need assistance I recommend you contact the maker of the program you use and ask them to reset the connection to BECU. Make sure they use the following information to download transactions:

      Org Name: BECU
      OFX ID: 1001
      URL: https://onlinebanking.becu.org

      If you have additional questions you can reply to this message. You can also contact us by phone during our business hours of Monday Friday 7:00 AM - 7:00 PM PST and Saturday 9:00 AM - 1:00 PM PST at 800-233-2328.

      Sincerely,

      Jeremy Nelson
      BECU Technical Support Analyst
      Phone 800-233-2328

      Delete
    3. If the above settings don't do the trick... if you use ofxVer: 103 for BECU, then you may need to delete the CLIENTUID line at the bottom of your sites.dat file. That's the only persistent "setting" kept by the scripts, other than the site and account settings.

      Delete
    4. BECU systems changed, but watching what Quicken does led me to these settings, which work.


      SiteName : BECU
      AcctType : BASTMT
      fiorg : becu
      fid : 325081403
      url : https://onlinebanking.becu.org/ofx/ofxprocessor.asp
      bankid : 325081403
      brokerid :
      appid : QWIN
      appver : 2300
      mininterval:
      ofxver : 103



      SiteName : BECU CC
      AcctType : CCSTMT
      fiorg : becu
      fid : 3670
      url : https://onlinebanking.becu.org/ofx/ofxprocessor.asp
      bankid : 325081403
      brokerid :
      appid : QWIN
      appver : 2300
      mininterval:
      ofxver : 103

      Delete
    5. Marvelous! I just saw this info on the Money dance site, and they mentioned using pocket sense to connect, too!
      KR

      Delete
  11. Just posting this here for reference, so I don't forget. Another (related) post from another site. See https://getsatisfaction.com/quickencommunity/topics/cant-download-discover-transactions-to-mac-quicken-2007-due-to-ol-226-error

    "I just got off chat with Discover. They stopped supporting Quicken 2007 as of Jan 2017. They then made some changes in mid-February 2017 (probably Feb 18 as Michael Smith mentioned earlier in this thread) which now keeps Quicken 2007 from working with Discover ... "

    ReplyDelete
  12. That was QWIN ver 1600 which I would have thought stopped working long ago. I think the Pocketsense default version is around 2300, so I don't think that is the issue at all. I've actually tried 2500 as well.

    ReplyDelete
  13. I have been playing with ofx.py.

    errmsg= "** An ERROR occurred attempting HTTPS connection to"
    h = httplib.HTTPSConnection(host, timeout=5)
    h.set_debuglevel(1)
    h.putrequest('POST', selector, skip_host=1, skip_accept_encoding=1)
    h.putheader('Host', host)
    h.putheader('Content-Length', str(len(query)))
    h.putheader('Content-Type', 'application/x-ofx')
    h.putheader('Accept', '*/*, application/x-ofx')
    h.putheader('Accept-Encoding', 'identity')

    errmsg= "** An ERROR occurred sending POST request to"
    p = h.endheaders(query)
    # p = h.request('POST', selector, query,
    # {"Content-Type": "application/x-ofx",
    # "Accept": "*/*, application/x-ofx"
    # }
    # )

    errmsg= "** An ERROR occurred retrieving POST response from"
    #allow up to 30 secs for the server response (it has to assemble the statement)
    h.sock.settimeout(30)
    response = h.getresponse().read()

    f = file(name,"w")
    f.write(response)
    f.close()

    Running this version of ofx.py in Setup.py option 7, test mode, only for my Discover accounts yields (private info rewdacted)

    DISCOVER BANK : ********** : Getting records since: 20170131
    send: 'POST / HTTP/1.1\r\nHost: ofx.discovercard.com\r\nContent-Length: 680\r\nContent-Type: application/x-ofx\r\nAccept: */*, application/x-ofx\r\nAccept-Encoding: identity\r\n\r\nOFXHEADER:100\r\nDATA:OFXSGML\r\nVERSION:102\r\nSECURITY:NONE\r\nENCODING:USASCII\r\nCHARSET:1252\r\nCOMPRESSION:NONE\r\nOLDFILEUID:NONE\r\nNEWFILEUID:************************************\r\n\r\n\n\n\n20170303103349\n**PRIVATE**\n**PRIVATE**\nENG\n\nDiscover Bank\n12610\n\nQWIN\n2500\n\n\n\n\n\n************************************\n4\n\n\n031100649\n**********\nCHECKING\n\n\n20170131\nY\n\n\n\n\n'
    reply: 'HTTP/1.1 403 Forbidden\r\n'
    header: Server: AkamaiGHost
    header: Mime-Version: 1.0
    header: Content-Type: text/html
    header: Content-Length: 269
    header: Expires: Fri, 03 Mar 2017 15:33:50 GMT
    header: Date: Fri, 03 Mar 2017 15:33:50 GMT
    header: Connection: close
    header: Set-Cookie: ...
    Invalid OFX statement.
    ** Review .\xfr\DISCOVERBANK20170303103349331974.ofx for possible clues...
    An online error occurred while testing the new account.

    Note, NO User-Agent header, no Expect header. Still get 403 response. These are the same headers sent by the original code, slightly different order.

    Here is hard-coding the query to match the one posted by robler on infinitekind. I hard-coded my user name, password, and account number in the query (redacted in output) as query = '...' just before the try:.

    DISCOVER : **************** : Getting records since: 20170131
    send: 'POST / HTTP/1.1\r\nHost: ofx.discovercard.com\r\nContent-Length: 720\r\nContent-Type: application/x-ofx\r\nAccept: */*, application/x-ofx\r\nAccept-Encoding: identity\r\n\r\n\r\n\r\n\r\n\r\n20170228232807\r\n**PRIVATE**\r\n**PRIVATE**\r\nENG\r\n\r\nDiscover Financial Services\r\n7101\r\n\r\nQWIN\r\n2500\r\n\r\n\r\n\r\n\r\n0\r\n\r\n\r\n**PRIVATE**\r\n\r\n\r\n20130101000000\r\n20170228232807\r\nY\r\n\r\n\r\n\r\n\r\n'
    reply: 'HTTP/1.1 403 Forbidden\r\n'
    header: Server: AkamaiGHost
    header: Mime-Version: 1.0
    header: Content-Type: text/html
    header: Content-Length: 269
    header: Expires: Fri, 03 Mar 2017 16:04:13 GMT
    header: Date: Fri, 03 Mar 2017 16:04:13 GMT
    header: Connection: close
    header: Set-Cookie: ...
    Invalid OFX statement.
    ** Review .\xfr\DISCOVER20170303110412992001.ofx for possible clues...
    An online error occurred while testing the new account.

    Sending his working OFX query still yields a 403.

    Until robler can post his/her header's, I don't think we will get anyplace.

    ReplyDelete
    Replies
    1. Formatting total messed up in second test. Changed all less than ... greater than to {...} so board doesn't strip them.

      DISCOVER : **************** : Getting records since: 20170131
      send: 'POST / HTTP/1.1\r\nHost: ofx.discovercard.com\r\nContent-Length: 720\r\nContent-Type: application/x-ofx\r\nAccept: */*, application/x-ofx\r\nAccept-Encoding: identity\r\n\r\n{?OFX OFXHEADER="200" VERSION="211" SECURITY="NONE" OLDFILEUID="NONE" NEWFILEUID="NONE"?}\r\n{OFX}\r\n{SIGNONMSGSRQV1}\r\n{SONRQ}\r\n{DTCLIENT}20170228232807{/DTCLIENT}\r\n{USERID}**PRIVATE**{/USERID}\r\n{USERPASS}**PRIVATE**{/USERPASS}\r\n{LANGUAGE}ENG{/LANGUAGE}\r\n{FI}\r\n{ORG}Discover Financial Services{/ORG}\r\n{FID}7101{/FID}\r\n{/FI}\r\n{APPID}QWIN{/APPID}\r\n{APPVER}2500{/APPVER}\r\n{/SONRQ}\r\n{/SIGNONMSGSRQV1}\r\n{CREDITCARDMSGSRQV1}\r\n{CCSTMTTRNRQ}\r\n{TRNUID}0{/TRNUID}\r\n{CCSTMTRQ}\r\n{CCACCTFROM}\r\n{ACCTID}**PRIVATE**{/ACCTID}\r\n{/CCACCTFROM}\r\n{INCTRAN}\r\n{DTSTART}20130101000000{/DTSTART}\r\n{DTEND}20170228232807{/DTEND}\r\n{INCLUDE}Y{/INCLUDE}\r\n{/INCTRAN}\r\n{/CCSTMTRQ}\r\n{/CCSTMTTRNRQ}\r\n{/CREDITCARDMSGSRQV1}\r\n{/OFX}'
      reply: 'HTTP/1.1 403 Forbidden\r\n'
      header: Server: AkamaiGHost
      header: Mime-Version: 1.0
      header: Content-Type: text/html
      header: Content-Length: 269
      header: Expires: Fri, 03 Mar 2017 16:04:13 GMT
      header: Date: Fri, 03 Mar 2017 16:04:13 GMT
      header: Connection: close
      header: Set-Cookie: ...
      Invalid OFX statement.
      ** Review .\xfr\DISCOVER20170303110412992001.ofx for possible clues...
      An online error occurred while testing the new account.

      Delete
    2. Just fyi... but I did ~ the same tests yesterday, with the same results.

      Delete
    3. Yes, that's a total bummer. Doesn't look easy now.

      Delete
  14. Finally had a little time yesterday to actually focus on the Discover issue. I cobbled together a small .net module to test connections using their http library. I was able to get a successful connection and ofx response, but I tested using OFX 2.x (xml), so still need to test with sgml. Initial (quick) comparisons between the python httplib debug output vs .net output didn't reveal anything obvious, so more to do...

    ReplyDelete
  15. This was posted by pgs over on ofxhome yesterday.

    Quoting:

    I was hitting the same thing, but thanks to robler's notes, I was able to get it working again as well. The server is very finicky about the order of the HTTP headers -- this sequence works for me:
    POST / HTTP/1.1
    Content-Type: application/x-ofx
    Host: ofx.discovercard.com
    Content-Length: [the length of the request]
    Connection: Keep-Alive
    With those headers, the XML request robler posted above works for me. Hope this helps.

    End Quote

    I was able to confirm that this order works in pocketsense for Discover Bank for both bank accounts and credit cards. This is still sending the normal ofxsgml headers. I made this change in ofx.py in function doQuery().

    ***** CAUTION: THIS HAS NOT BEEN FULLY TESTED WITH OTHER BANKS *****
    Let Robert supply a tested version that he confirms works with other banks also.
    *********************************************************************
    try:
    errmsg= "** An ERROR occurred attempting HTTPS connection to"
    h = httplib.HTTPSConnection(host, timeout=5)
    # Uncomment next line to turn on debugging info
    # h.set_debuglevel(1)
    h.putrequest('POST', selector, skip_host=1, skip_accept_encoding=1)
    h.putheader('Content-Type', 'application/x-ofx')
    h.putheader('Host', host)
    h.putheader('Content-Length', str(len(query)))
    h.putheader('Connection', 'Keep-Alive')

    errmsg= "** An ERROR occurred sending POST request to"
    p = h.endheaders(query)
    # original code
    # p = h.request('POST', selector, query,
    # {"Content-Type": "application/x-ofx",
    # "Accept": "*/*, application/x-ofx"
    # }
    # )

    errmsg= "** An ERROR occurred retrieving POST response from"
    #allow up to 30 secs for the server response (it has to assemble the statement)
    h.sock.settimeout(30)
    response = h.getresponse().read()

    f = file(name,"w")
    f.write(response)
    f.close()

    ReplyDelete
    Replies
    1. Excellent. I actually went down a different path when I started focusing on the issue. Discover appeared to be fully supporting OFX 2.x for direct-connect, so I assumed it would eventually become the "standard". That may not happen, but I started adding support for OFX 2.x messages (xml), and got that working. The catch is that Money doesn't support OFX 2.x, *but* the difference between sgml and xml is minor. Replacing the message header with a 1.x version gets past the Money import "format check", and the extra trailing field closures are accepted (although optional) in sgml. Seems to work fine.

      I had assumed that I'd either be adding a compiled module for the http connection, or writing a sockets interface in the scripts (for full control). If the manual httplib request you've used works reliably, then that's even better!

      Thanks Andrew.

      Delete
    2. Ditto Andrew - I made 2 setup.py/ofx.py combos for Discover/non-Discover accounts and am back in business (I prefer downloading per-account rather than in a batch). MUCH appreciated for both of y'all's work!

      Delete
    3. From another post by pgs:

      Quoting:

      The key was robler's comment about the required options for the .NET HttpWebRequest class -- once I got that working I was able to log the HTTP request. It's so sensitive that I wasn't even able to get the http library I was previously using to work and ended up writing a custom client just for this server.

      End quoting

      pgs was able to see the debug the http headers in .net and found the order that worked. This despite the standard stating the header order should not matter for unique headers!

      I took a look at the generated ofx responses.
      Discover Card is still broken the same like {FITID}FITID20170216408.850 after scrubbing.
      Discover Bank is still broken the same like {FITID}SDF7957870 after scrubbing.

      Why is it that Discover is just so incompetent.

      Delete
    4. **** CAUTION ****
      I strongly urge that if you don't need to download Discover transactions today, you should wait until Robert has an official release.
      **** CAUTION ****

      NOTE: The previous post was deleted because a debugging line that might change the behavior was left inadvertently.

      This should be a safer change for the doQuery() function in ofx.py. It explicitly treats Discover stuff separately. Note that if you want to try it, you need to correct the indentation which is stripped by the board.


      def doQuery(self,query,name):
      # urllib doesn't honor user Content-type, use urllib2
      url = FieldVal(self.site,"url")

      isDiscover = 0
      if 'DISCOVERCARD' in url.upper():
      isDiscover = 1

      garbage, path = urllib2.splittype(url)
      host, selector = urllib2.splithost(path)
      response=False


      try:
      errmsg= "** An ERROR occurred attempting HTTPS connection to"
      h = httplib.HTTPSConnection(host, timeout=5)
      # h.set_debuglevel(1)

      errmsg= "** An ERROR occurred sending POST request to"
      if isDiscover:
      h.putrequest('POST', selector, skip_host=1, skip_accept_encoding=1)
      h.putheader('Content-Type', 'application/x-ofx')
      h.putheader('Host', host)
      h.putheader('Content-Length', str(len(query)))
      # h.putheader('Accept', '*/*, application/x-ofx')
      # h.putheader('Accept-Encoding', 'identity')
      h.putheader('Connection', 'Keep-Alive')
      p = h.endheaders(query)
      else:
      p = h.request('POST', selector, query,
      {"Content-Type": "application/x-ofx",
      "Accept": "*/*, application/x-ofx"
      }
      )

      errmsg= "** An ERROR occurred retrieving POST response from"
      #allow up to 30 secs for the server response (it has to assemble the statement)
      h.sock.settimeout(30)
      response = h.getresponse().read()

      f = file(name,"w")
      f.write(response)
      f.close()
      except Exception as inst:
      self.status = False
      print errmsg, host
      print " Exception type:", type(inst)
      print " Exception Val :", inst
      if response:
      print " HTTPS ResponseCode :", response.status
      print " HTTPS ResponseReason:", response.reason

      if h: h.close()

      Delete
  16. I was late getting home, but had time to test the updates done thus far. In a nutshell, the new version builds the ofx header using the rules posted by Andrew above, and worked with all of the sites that I personally use. Additionally, I added support for OFX 2.x, and successfully tested with Discover. Actually, I tested Discover with ofxVer=102, 103, and 211, and all worked fine w/ Money. Discover bank downloads appear to be offline at the moment, so I couldn't test import to Money, but connections worked fine earlier when I first wrapped up the script edits.

    I've posted a beta version @ https://sites.google.com/site/pocketsense/ofxpy_pocketsense_beta.zip . If anyone has time to test using their account settings, that would be helpful. I'd recommend stepping through each individually using the Setup module, rather than all at once with Getdata.

    ReplyDelete
    Replies
    1. All of mine worked flawlessly, including Discover Card and Bank:

      TDAMERITRADE
      USAA FEDERAL SAVINGS BANK (Checking)
      USAA FEDERAL SAVINGS BANK (Savings)
      USAA CC (Amex)
      E-TRADE INVESTMENTS
      OPTIONSXPRESS
      DISCOVERBANK
      DISCOVERCARD
      CHASE CC
      CITIBANK COSTCO

      Thanks so much!

      Delete
    2. I have been using Pocket Sense ever since Money abandoned us. If I am techie at all it would be level 1, but I somehow managed to set it up reading and rereading instruction. I have not a clue what most of this discussion means, but I knew Discover screwed it up and wouldn't help, however I took a chance and downloaded the beta file and managed to set it up again (a long time since I set it up first time). It works with my Citi and my Discover. I don't know why, but thank you, thank you very much is all I can say. Rette

      Delete
  17. Hi Robert ... Just tested your latest zip and it works fine on all my Institutions just like the current version does.

    One new wrinkle just showed up today.....Citibank Fails with INVALID OFX both current version AND new beta version. I am able to download with win phone version of my software, but not pocketsense. I suspect something Citi changed over the weekend as everything worked before Monday morning.

    Looks like banks are making changes to keep us occupied.

    Raymond

    ReplyDelete
  18. Scratch my previous message about Citibank....It also works fine with the beta version of pocketsense. The current version does not work.

    Raymond

    ReplyDelete
    Replies
    1. Using the current non-beta version, I am also now failing with a 403 error with Citibank.

      Harold

      Delete
  19. Robert,

    Any chance you can fold in the siteClientUID changes previously proposed as well? I've been running Pocketsense with those changes manually added just in case Schwab ever switches to ofx 1.03 and I have to keep manually adding the changes in for each revision. It's a nice revision for those that have multiple accounts at one ofx 1.03 institution.

    thanx,
    ameridan

    --------------------------------------------------------------------------------
    --- OfxPy/orig/ofx.py Wed Feb 11 10:07:36 2015
    +++ OfxPy/ofx.py Thu Feb 11 23:37:36 2016
    @@ -86,7 +86,14 @@
    clientuid=""
    if "103" in self.ofxver:
    #include clientuid field only if version=103, otherwise the server may reject the request
    - clientuid = OfxField("CLIENTUID",userdat.clientuid)
    +
    + # if a site-level clientUID is defined, use it, otherwise default to global clientUID
    + clientuidval = userdat.clientuid
    + if FieldVal(site,"siteClientUID") <> '':
    + clientuidval = FieldVal(site,"siteClientUID")
    +
    + clientuid = OfxField("CLIENTUID",clientuidval)
    + print "using ClientUID of: ", clientuidval

    fidata = [OfxField("ORG",FieldVal(site,"fiorg"))]
    fidata += [OfxField("FID",FieldVal(site,"fid"))]

    --- OfxPy/orig/site_cfg.py Wed Feb 19 20:51:56 2014
    +++ OfxPy/site_cfg.py Thu Feb 11 23:11:21 2016
    @@ -142,6 +142,7 @@
    bankid=''
    brokerid=''
    ofxver = '102'
    + siteClientUID = ''
    appid = DefaultAppID #defined in control2.py
    appver = DefaultAppVer
    mininterval = 0
    @@ -161,6 +162,7 @@
    'BANKID': bankid,
    'BROKERID': brokerid,
    'OFXVER': ofxver,
    + 'SITECLIENTUID': siteClientUID,
    'APPID': appid,
    'APPVER': appver,
    'MININTERVAL': mininterval,
    @@ -181,6 +183,7 @@
    elif field == 'BANKID': bankid = value
    elif field == 'BROKERID': brokerid = value
    elif field == 'OFXVER': ofxver = value
    + elif field == 'SITECLIENTUID': siteClientUID = value
    elif field == 'APPID': appid = value
    elif field == 'APPVER': appver = value
    elif field == 'MININTERVAL': mininterval = int(value)

    ReplyDelete
  20. Hi Dan: Can you refresh my memory on this one? I recall adding support for multiple accounts at one institution, where they all have the same username. Is this related, or different?

    ReplyDelete
    Replies
    1. I think Dan may be referring to the issue that came up at Chase that forces us to run two instances of Pocketsense because of some GUID (or something like that) field?

      Harold

      Delete
  21. This is that issue, but I don't recall you actually incorporating it. Here's the link to the author's site to jar your memory: http://pastebin.com/9UJ333RZ

    Do you think ofx 2.0 will be next for some of these financial institutions, rather than 1.03? If so, I guess we could hold off till it actually happens, but I thought the author (forget who) had come up with a nice solution.

    ------------------------------------------
    By the way I just tested your Beta at Schwab, and Money didn't like it "The file you attempted to import appears to be invalid..."

    Here's my output file header:
    OFXHEADER:100

    DATA:OFXSGML

    VERSION:2500

    SECURITY:NONE

    ENCODING:USASCII

    CHARSET:1252

    COMPRESSION:NONE

    OLDFILEUID:NONE

    NEWFILEUID:dd1a870f-23a5-47dc-bae7-30ec160c1139

    vs. my file header from Friday:
    OFXHEADER:100
    DATA:OFXSGML
    VERSION:102
    SECURITY:NONE
    ENCODING:USASCII
    CHARSET:1252
    COMPRESSION:NONE
    OLDFILEUID:NONE
    NEWFILEUID:NONE

    where did version 2500 come from?

    ReplyDelete
  22. Raymond, my Citibank stopped working today too. I got rid of my beta before trying because it didn't work for Schwab. What are these banks doing?

    ReplyDelete
  23. I set the default version to 2500... to see if it "broke" anything. Guess it did :) I wondered if servers did a "greater than or equal to", or looked for specific values. Try Schwaab w/ a lower value, and let me know.

    ReplyDelete
    Replies
    1. The beta worked for Schwab; it's Money that didn't like the file and that was the only difference I could find, But I substituted the entire header in today's ofx file and Money still rejected it, so the version wasn't the problem.

      Delete
    2. The non-beta download works for Schwab? If so, will need to compare the message body, since I don't think I changed anything in the request, except when ofxVer=2xx.

      Delete
    3. Robert,
      You have ofxver set to 2400. I had 2500 in my sites.dat file with my current setup and it works.

      Delete
    4. I think I'm confused. I set default appVer=2400, but ofxVer should default to 102 if not specified in sites.dat. I missed that above (appVer vs ofxVer).

      Does Schwab work w/ the beta when ofxVer=102 and appVer=2500?

      Delete
    5. Excellent catch Robert!!
      I apologize ( I was confused too seeing VERSION:2500). I was experimenting in December and mistakenly entered 2500 in my sites.dat file for Schwab ofxVer, rather than appVer. Until the beta, it was ignored I guess. All is well now with Schwab and your beta. YEAHHH

      Delete
    6. And Citi and Discover and PNC all work too! Great job.

      Delete
    7. That's a relief :) Had me worried. Thanks Dan!

      Delete
  24. I found the discussion re: possibly adding site-specific ClientUID values, and remember why I didn't do anything. It's one of those "easy to do but messy" hacks. If I'm understanding, the underlying issue is that each username needed it's own UUID, or more likely, the bank is associating a UUID with a username, and doesn't allow it to be reused. I tossed out the idea of adding a site-specific ClientUID field, so a user can have multiple copies of a site, and use one for each unique username. It was an easy fix from a code perspective, but messy to use and not well understood, so I left it alone. What I hoped to learn (but didn't) was whether a site required a username to have the same UUID across all accounts, or if it was ok to assign unique UUIDs per account setting. That would be an automatic (elegant) solution.

    ReplyDelete
  25. Hi All,

    I just tried the beta with Discover Credit Card.

    I'm getting the following error when testing it with Setup.py:

    ** An ERROR occurred sending POST request to ofx.discovercard.com
    Exception type:
    Exception Val : endheaders() takes exactly 1 argument (2 given)
    An online error occurred while testing the new account.

    -Kevin N.

    ReplyDelete
    Replies
    1. The board remove the text after the Exception type: probably due to the less than sign and the greater than sign.

      Exception type:(less than sign) type 'exceptions.TypeError' (greater than sign)

      Delete
  26. Kevin, did you overwrite control2.py, ofx.py and rlib.py? I got that message thinking that all the changes were in ofx.py but apparently the other 2 files are needed too.

    ReplyDelete
  27. Hi NexTek,

    I extracted the new beta zip file to a new empty folder.
    Ran Setup.py
    Closed Setup.py
    Ran Setup.py again
    Added my Discover card account and tested it.

    That's when I got the error

    -Kevin N.

    ReplyDelete
    Replies
    1. I replicated your steps, and it worked for me. Before tracing out differences, what version of Python are you running? I tested using Python 2.7.12 and 2.7.13.

      Delete
    2. Hi Robert,
      I'm using Active Python 2.6.5.12
      It looks like 2.7.13 is available at Active, I'll install that and give it a try.

      Thanks for getting back to me. I'll let you know how it goes.

      -Kevin N.

      Delete
    3. Hi Robert,

      2.7.13 did the trick! Discover Card works as expected.

      Thank you.

      -Kevin N.

      Delete
  28. Hi NexTek,

    Have you tried John's changes to ofx.py and site_cfg.py for separate ClientUID's with the new beta?

    -Kevin N.

    ReplyDelete
    Replies
    1. No, I don't actually have a way to test it; I just added it to my previous version thinking that soon ofx 1.03 would be implemented by more financial houses. Like Robert implied, it is easy for someone that has a need for it to add, rather than having separate Pocketsense implementations for each additional account.

      Delete
    2. Hi NxtTek,

      OK. I need the changes for His and Her's Chase credit card accounts. I'll give them a whirl and see how it works.

      -Kevin N.

      Delete
    3. I kinda' wish Robert would add the code, as it doesn't get in the way at all if not needed. Unless he has a more "elegant" solution. ;)

      Here's my blog article on implementing it for others to reference. https://microsoftmoneyoffline.wordpress.com/2016/04/26/resolving-errors-for-those-having-multiple-accounts-at-chase-and-other-ofx-103-banks/

      Delete
    4. I don't mind adding it, and it's easy to do. I originally had thoughts of automating, creating unique uuid values for each client/username/account combo, but wasn't sure if the uuid should be the same for multiple accounts under the same username? The idea was to create a (hashed) lookup table of unique pairs, adding new ones as they come along. It wouldn't be human "readable", but there wouldn't be any extra config.

      Delete
  29. Hi Robert, everyone -

    I downloaded and setup the beta version but am still receiving the same error. I did install Python 2.7.13 (from 2.7.5) and still no luck:

    ** An ERROR occurred sending POST request to ofx.discovercard.com
    Exception type: (less than sign)class 'socket.error'(greater than sign)
    Exception Val : {Errno 10061} No connection could be made because the target machine actively refused it.

    Ben

    ReplyDelete
    Replies
    1. Looks like you have a different issue. Compare your sites.dat entry for Discover to the following:

      SiteName : DISCOVER
      AcctType : CCSTMT #credit card
      fiorg : Discover Financial Services
      fid : 7101
      url : https://ofx.discovercard.com
      bankid :
      brokerid :
      ofxVer : 103
      appid :
      appver :
      mininterval:
      timeOffset :

      Delete
    2. Thanks for the follow-up Robert! My sites.dat entry was missing the ofxVer: 103 line - I added it but the error persists. :(

      Delete
    3. Is it only happening with Discover (i.e., do other sites work)?

      Delete
    4. Robert, I'll keep the ofxVer in mind, but mine worked today leaving that blank.

      Delete
    5. You are correct... but I knew the settings above worked on mine. I tested with 102, 103, and 211 yesterday.

      Delete
    6. Hi Robert - was out of town the last couple of days so I apologize for the delay in my response. To answer your question, yes, it only happens with Discover - I have it setup for AMEX and it works fine.

      Delete
    7. Hi Robert - out of curiosity, I tried 102 as the value for ofxVer and it works! Wahoo! Thank you!

      Delete
  30. Hi Robert,

    I just want to confirm that the new beta works with:

    Chase Credit Cards - 2 accounts (with John's changes)
    Discovercard - 2 accounts (with ofxVer 103)
    Regions Bank - 4 accounts
    Fidelity Investments - 4 accounts

    Thank you again for your continued work on the scripts.

    -Kevin N.

    ReplyDelete
  31. Regarding the use use "user specific" ClientUID values, before adding the site->clientUID option, I'd like to understand what we're trying to fix. I know that some folks are using a custom UID value for each username (his and hers, if you will), but which of the following requirements is true:

    a) a unique UID value is needed for each site+username combo
    b) a unique UID value is needed for each site+account combo, or
    c) a unique UID value is needed for each site+username+account combo

    Anyone know?

    ReplyDelete
    Replies
    1. I believe that for Chase, it's site+username.

      Harold

      Delete
    2. Hi Robert,

      In my case with Chase Credit Cards (his and her accounts) Obviously, the site is the same, the account numbers are different and thus the usernames and passwords are different.

      The (hers) account uses a unique siteClientUID.
      The (his) account uses the global ClientUID
      Both use ofxVer 103

      I retained the siteClientUID and the global ClientUID from my prior version of the scripts. In the past, changing the ClientUID triggered a prompt from Chase to verify my ID at their website. I'm thinking that a new or different ClientUID appears as a new computer / connection and they need to add that connection as a trusted connection.

      HTH -Kevin N.


      Delete
    3. Since Fidelity (and maybe others) allow the same account number for mulitple usernames, I'm going to say that keying to the site and username is the safest bet. My preference at this point will be to simply add unique clientUID values based on site+username, and store them in a lookup table/file. The site and username would be hashed, for security. This approach would be fully automatic, with nothing special by the user. If it turns out we need site+user+account lookup, then it's easy to add. Sound ok?

      Delete
  32. Sounds good to me Robert. Sounds like Kevin may have to make another verification call for their Chase accounts (as an example) unless the table allows overriding the automatically generated clientUID values with their existing ones.

    ReplyDelete
    Replies
    1. I wondered about that. Kevin: Do they require verbal confirmation, or is just an email?

      Delete
    2. This was a while ago now, but I think that Chase just requires a different UID value...

      Harold

      Delete
  33. Hi Robert,

    It seems that when Chase detects a new computer / connection it triggers a notification email that requires the user to log onto the Chase website. Then, through a secure message, the user clicks on a 'Verify your identity' link.

    My only concern is that they [Chase] may limit the number of trusted connections to any given account. I'll look further into that on their site.

    -Kevin N.

    ReplyDelete
    Replies
    1. Hi Robert,

      I sent a message to Chase regarding the number of trusted connections / computers that can be assigned to any given account.

      I won't bore you with the somewhat boilerplate response. Essentially, all that is needed is to verify my identity with each new computer / connection / ClientUID.

      If I understand you correctly, your plans are to create a separate table for ClientUIDs. Can that table remain intact with subsequent upgrades to the scripts? If so, then that would be ideal.

      -Kevin N.

      Delete
    2. My intent was that the ClientUID values would be reused indefinitely for a matching url + username.

      Delete
    3. Brilliant! Thank you for your consideration.

      Delete
  34. Just downloaded the beta. It seems like Discover worked fine, as did every other account including Chase.

    ReplyDelete
  35. Robert, Looking at the Beta reminded me of two scrubber issues.

    The first had to do with Discover and the strange $0.00 "second line in statement" transactions that it downloads. Did you ever come up with a scrubber for that? I remember reading about it years ago.

    Second we had discovered that some 401k downloads (Net Benefits, for instance) would put the wrong sign in front of reinvestment transactions, and Money would ignore the transaction. I wrote a quick scrubber for it a while back. Are you interested in incorporating it to the main trunk?

    Carlos

    ReplyDelete
    Replies
    1. 1. I had decided not to ignore the $0.00 transactions, since they usually contain additional transaction info. I'm not opposed to adding a filter option, but that was how I'd left it before.
      2. Pretty sure the "wrong sign" issue has been addressed in scrubber.py (?).

      Delete
    2. Hmm. Most of my $0.00 transactions usually have some random, 11-character, meaningless code. Not a big deal either way. I could see how sometimes the $0.00 transaction may be useful

      The version of scrubber.py I have shows a change by you from 2013 that addresses INVBUY or INVSELL transactions. From the comments I made in the file year+ ago, it looks like I copied your_scrubINVsign() functions to create a new function that fixes REINVEST transactions.

      And once again, thanks for all the work you've done with this. It's amazing we are still able to use MS$ eight years after it went off the market.

      Delete
    3. Should have said "some" have extra info, but you're right. Most do not. I may play around with it.

      Just to clarify, we're only talking about adding transactions with a REINVEST type?

      Delete
    4. Robert,

      Sorry, I wasn't checking in for a while.

      On Discover: yes, some transactions have extra info, and many do not. Contrary to what NxtTek say, these are not pre-authorizations, as they can show up days after the related transaction. They are some "extra" info that Discover includes in their statements, like cash back, but for some reason when you download they just show up as a meaningless code.

      On Fidelity: Yes, I have an equivalent fix for REINVEST transactions to the one you have for INVBUY and INVSELL.

      Delete
  36. "$0.00 transactions usually have some random meaningless code."
    These are pre-authorization transactions, meaning a real transaction will probably be occurring the next day. Since I download on a daily basis, their existence can be informative for fraud detection if you haven't ordered anything.

    ReplyDelete
    Replies
    1. On Discover, I've found that it sends a $0.00 transaction only when it is a transaction that is eligible for their increased cashback offer for that quarter. Citibank occasionally sends a $0.00 transaction, but it is usually for something like a "membership charge" which my account doesn't actually get charged. None of my other banks sends $0.00 charges.

      Harold

      Delete
  37. Hi, Robert... if you're considering including minor improvements, the change that I've made locally that I find useful is retrying a bad HTTP return code 5 times. For each time, I put out a line telling what the return code I got was. (Even with this 403 thing that is coming back now from Discover and Citibank, it retried five times before giving up.) I can't tell you why, but various banks occasionally come back with a bad HTTP return code, and just retrying it makes it work fine. Chase does this about once a month, and other banks seem to do it, too, from time to time.

    ReplyDelete
    Replies
    1. I'd be ok with specific retry reasons, but there are legitimate causes where I probably wouldn't want it retrying over again (e.g., bad url). Do you have some specific urllib exceptions that you see?

      Delete
    2. I've gotten all kinds of responses over time, including some that made no logical sense. So I check response.status... if it's anything but 200, I go to sleep for a second, close the connection, and loop back to re-establish the connection. And I let the loop run five times before giving up. It has made the process more "robust" for me, for lack of a better word.

      Delete
  38. I've posted a new beta version @ https://sites.google.com/site/pocketsense/ofxpy_pocketsense_beta.zip .

    A summary of changes:
    - Changed clientUID to be specific to the user and bank url. It's "automatic", with no special user config. Note that the first time you connect, a new clientUID will be created, but then reused indefinitely, unless reset by the user via Setup (see below).
    - Modified Setup.py to allow "resetting" an account connection (clears clientUID for user@site). Deleting an account entry also resets the connection.
    - Added skipZeroTransactions: Yes/No option to sites.dat. If Yes, $0.00 transactions are removed from statements. Default=No
    - Added REINVEST transaction types to _scrubINVsign()

    Feel like I'm forgetting something, but I think that's the major edits for now, plus a few bug fixes. Feedback appreciated.

    ReplyDelete
    Replies
    1. Hi Robert,

      I just wanted to confirm with you that the 03-12-2017 beta version of the scripts work as expected with:
      Chase Visa
      Discovercard
      Fidelity
      Regions Bank

      The 'Test' using Setup.py for Chase *did* initially throw an error until I verified my identity @ Chase.
      The $0.00 amount scrubber seems to also work as expected with Discovercard.
      I tried the new 'Delete or Reset' option in Setup, up to the point of where I could (C) Cancel.

      I'd say that you've hit another home run. Well done!

      I can't thank you enough for your work on the scripts.

      -Kevin N.

      Delete
    2. Hi Robert,

      I meant to add that there is no longer any need for 'His' and 'Hers' < site > entries in sites.dat with regards to having more than 1 account @ Chase.

      Again, well done!

      -Kevin N.

      Delete
  39. Worked with all but my citi accounts but I think citi discontinued their OFX recently as I couldn't get it lately with old version.
    Worked with: Chase, Discover, CapOne360, Vanguard, AmexStarwood
    A note on Chase:: I had to re-authorize it after first run...get email, go to secure message and follow instructions.

    If anyone can confirm...is Citi OFX dead..might be time to kill a credit card or two.

    ReplyDelete
    Replies
    1. My bad. Citi is working fine. Not sure what was going on but currently on the beta it is working great. Thank you for all your work...now if only MS would open source MSmoney.

      Delete
  40. Great job Robert!!

    Worked fine for all my accounts. Since you're looking for feedback thinking you forgot something, the only very minor things I noticed:
    1) last line of Revisions in sites.dat:
    # -Added skipZeroTransactions

    2) your code still addends generic clientUID to sites.dat which may very well be intentional

    Thank you so much for bringing up-to-date.

    ReplyDelete
  41. Works fine here, Robert. Including AT&T (run by Citicard and stopped working with last full release version a few weeks ago, Chase (required an online verification), Fidelity, and others for which I have not seen reports of issues here. Thank you for continuing to support this invaluable add-in.

    ReplyDelete
  42. First, thank you so much for continuing to support PocketSense! It's awesome being able to still use Microsoft Money after all these years.

    I just upgraded to the new beta after my Citi credit card transactions stopped downloading with the older versions of PocketSense. That fixed the problem and everything is working great.

    However, I did find one small bug in the beta. I tried removing an account and I deleted it from the sites.dat file before removing it from PocketSense using setup.py. When I went to remove it from PocketSense, it would crash and close without removing it. To get it to remove successfully I had to put the site back in my sites.dat file and remove it in PocketSense first, before deleting it from the sites.data file, again.

    ReplyDelete
  43. sites.dat holds generic sites data, not account data. Your account data refers to the sites data, hence the crash.

    ReplyDelete
  44. Thanks for the feedback guys. A couple of notes:
    - I left the clientUID in sites.dat in case someone reverted to an earlier version... and it isn't used by the new version, so no harm, no foul.
    - I just now modified Setup to only delete the connection key if the site still exists in sites.dat. That corrects the crash "bug", but... if a person deletes the site before clearing the account, there will be dangling connection keys in the connections file (connect.key), since it can't delete the key without knowing the url. Not recommended ;)

    ReplyDelete
  45. I need to upgrade for Citibank, not sure I want to.
    Does this mean We can no longer control what the ClientUID is and have to accept whatever is generated? and how do we get it's value. I need the value for other software so I don't have to go through the new clientUID routine each time.

    Raymond

    ReplyDelete
    Replies
    1. I'll add an option in Setup when listing accounts to include the clientUID in the list. Once created, it won't change. They are stored in connect.key. The username and url are "hashed", but the clientUID is not.

      Delete
  46. The beta originally seemed to work fine for me. I had to re-register both Chase and Capital One, but it was easy enough, and the new code allowed me to get rid of my two PocketSense directories. Citibank and Discover both initially seemed to work well. However, oddly enough, when I ran the full download, my Discover Bank entries, which I had entered manually earlier in the month, did not make me go through the normal Money sequence of matching downloaded transactions to my manually entered transactions... it also showed in the account register as unmatched. However, oddly, if I ran Setup and Test Account, it worked fine, even if I did more than one in the same run of Setup. (I left myself two more Discover Bank entries to test a lter version of Pocketsense.)

    ReplyDelete
    Replies
    1. I have no idea how to code in Python nor do I understand OFX "stuff". However, I retrofit my old changes to your beta ofx.py DoQuery routine to retry five times in case of an invalid HTTP response code. I noticed that the old code used to have a statement that said "if h: h.close()" and the new code doesn't have that... I have no idea what it was supposed to do, nor if it was simply unnecessary, but in light of the post above, I thought I'd just point that out.

      Delete
  47. Harold, my Discover transactions are all matching up fine. Perhaps you downloaded some from the web? Those didn't match up for me after the beta trial, I think because the scrubber routine makes some changes to the FID (or datestamp?). I had to manually match those that I had downloaded from the web, but all is well again now.

    ReplyDelete
    Replies
    1. I entered them manually. And it seems really weird to me that it works and matches up when I am running from Setup, but it seems to ignore the downloaded transactions when I run Getdata. It certainly seems like it might be the scrubber stuff, but why should that make a difference?

      Delete
    2. It is odd, since the process is identical... *unless* you enabled CombineOFX, and are packaging all statements into one. If so, I'm guessing that Money is somehow interpreting it differently. I use Discover also, but never entered anything manually during the "outage", and I do use CombineOFX. No issues since resuming downloads.

      Delete
    3. So keep in mind that I don't understand this stuff except generally. I tried adding the h.close statement, but that didn't eliminate the problem. Then I tried deleting the transaction, and adding it back to Money manually, and I got the same behavior... it didn't work with Getdata but it did work with Setup.

      So then I looked at the downloaded files, and lo and behold the one from Getdata did not have the transaction and the one from Setup did.

      The obvious clue that I can think of is that the DTSTART and DTEND values for the file that came as a result of Setup were 2/11/2017 and 3/1/2017, while the values for the file that came as a result of Getdata were 2/28/2017 and 3/02/2017. The problem transaction was a end-of-month transaction with a posting date of 2/28/2017.

      I have no idea how or why these dates were set.

      I am NOT running CombineOFX. I am running with defaultinterval defaulting to 14, with no overrides. The transactions are for Discover Bank.

      Harold

      Delete
    4. I think I "got it" almost as soon as I hit enter on my prior post. Today is March 15th, so with defaultinterval=14, I should NOT get the transaction from 2/28. I suspect that Setup uses a different logic to specify the start date for transactions and that's why I got the transaction when using the Setup test function.

      H

      Delete
    5. I believe you do. I misunderstood the original issue, thinking you were getting *duplicate* downloads w/ Getdata, that weren't matching with originals. The download interval for testing in Setup is 31 days, w/ the hope of catching at least one transaction over the previous month.

      Delete
  48. Some minor bug fixes I noticed, and added the option to show clientUID (connection key) when listing accounts in Setup. I think it's about ready to put out as "final", but that doesn't mean other ideas won't be considered for adding (e.g., retries, etc.).

    https://sites.google.com/site/pocketsense/ofxpy_pocketsense_beta.zip

    ReplyDelete
    Replies
    1. Not that you even remotely need my code for the retries, but here's my version of DoQuery with the before-newest beta:


      # This routine has been changed to retry in case
      # an error response is received for the query.

      RETRY_INTERVAL = 1 # Retry every n seconds
      RETRY_TIMEOUT = 5 # Max number of seconds to retry
      DOWNLOAD_SUCCESS = 0

      RetryTime = 0 # seconds already retried


      # urllib doesn't honor user Content-type, use urllib2

      response=False

      while RetryTime < RETRY_TIMEOUT:
      try:
      errmsg= "** An ERROR occurred attempting HTTPS connection to"
      h = httplib.HTTPSConnection(self.urlHost, timeout=5)
      if Debug: h.set_debuglevel(1)

      errmsg= "** An ERROR occurred sending POST request to"

      h.putrequest('POST', self.urlSelector, skip_host=1, skip_accept_encoding=1)
      h.putheader('Content-Type', 'application/x-ofx')
      h.putheader('Host', self.urlHost)
      h.putheader('Content-Length', str(len(query)))
      h.putheader('Connection', 'Keep-Alive')
      p = h.endheaders(query)

      errmsg= "** An ERROR occurred retrieving POST response from"
      #allow up to 30 secs for the server response (if it takes longer, something's wrong)
      h.sock.settimeout(30)
      response = h.getresponse()
      if response.status == 200: # Good response code
      RetryTime = RETRY_TIMEOUT + 1 # Terminate loop
      respDat = response.read()

      #if this is a OFX 2.x response, replace the header w/ OFX 1.x
      if self.ofxver[0] == '2':
      respDat = re.sub(r'<\?.*\?>', '', respDat) #remove xml header lines like
      respDat = OfxSGMLHeader() + respDat.lstrip()

      f = file(name,"w")
      f.write(respDat)
      f.close()
      DOWNLOAD_SUCCESS = 1 # Set flag
      else:
      RetryTime += RETRY_INTERVAL
      print " +Error response from server:", response.status, " Retry time:", RetryTime
      time.sleep(RETRY_INTERVAL)
      h.close()


      except Exception as e:
      RetryTime = RETRY_TIMEOUT + 1 # Terminate loop
      self.status = False
      print errmsg, self.urlHost
      print " Exception type :", type(e)
      print " Exception val :", e

      if response:
      print " HTTPS ResponseCode :", response.status
      print " HTTPS ResponseReason:", response.reason

      if DOWNLOAD_SUCCESS == 0: print " +Error downloading file"

      Delete
    2. OK... OK... so did I know that this posting board would get rid of all indentation, which is relevant in Python? I don't think there is a way for me to delete a message, or I would delete my sample code.

      H

      Delete
    3. Robert,
      Many thanks for keeping this invaluable add-in functional. I ran into a problem with my AT&T card (a CitiCard brand) which was fixed by this latest beta. My only problem is the few institutions that don't have OFX servers.

      Delete
    4. I took the plunge and ran through the beta. It all worked fine for Citibank, BankofAmerica,Chase, and Fidelity.

      As far as Chase is concerned, I was able to edit the connect.key file and did not have to mess with a new ClientUID routine @ Chase.

      Still would be interested to know what dot Net function you used to test with.

      Raymond

      Delete
    5. Harold: I took a few minutes to look at adding a retry option, but I can't test, and I'm not sure what conditions you've had where it makes it to the response check (except an actual response). The only issue I've seen (and can reproduce) are timeouts, which throw an exception.

      Delete
    6. I hear you, Robert. Even since putting in the beta, I've been having good days lately, with no errors. If you want to test, though, you can just go back to the version of Pocketsense before the beta and try Discover. Of course, the 403 error will turn out to be non-recoverable.

      OTOH, it's really not a big deal. When I get your code, it typically takes me just a few minutes to integrate my changes.

      Thanks a lot for listening, and for keeping this great software going!

      Harold

      Delete
  49. Same here Ron. How do we get PayPal do get on board? :-)

    ReplyDelete
  50. Just tried the latest beta. Observations:
    - All accounts connected fine
    - I did have to re-register with Chase, which interestingly it was not required with the last beta.
    - The Discover $0.00 scrubber works great. Since it is a global setting, rather than an account setting, I'll keep an eye out to make sure there are no unexpected impacts to other accounts.
    - Cannot test the REINVEST sign issue until there are new REINVEST transactions ready to come down.
    - A small handful of transactions in two different accounts are not getting matched. Even after I deleted the old downloaded copy and accepted the new downloaded copy. I will have to wait a day or two to see how widespread the problem is.

    Carlos

    ReplyDelete
  51. Weird Chase issue this (Sunday 3/19) morning.... with next to last beta... CHASECREDIT returned {CODE}2000{SEVERITY}ERROR{MESSAGE}ofx.validationException] sporadically (I replaced angle brackets with braces to allow it to post). It would come back this way... then work fine... then come back this way.... unpredictable. It worked fine last night and last week...

    Harold

    ReplyDelete
    Replies
    1. Hi Harold,

      Sunday mornings is when many financial institution run general maintenance on their servers. Do you think *that* might be the cause of the anomaly?

      I just ran the 12-Mar-2017 version of the scripts without incident re Chase.

      -Kevin N.

      Delete
    2. Hi Harold,

      After running the scripts, I ran Moneydance, which also uses OFX direct-connect protocols and it threw the same error that you reported.

      The fact that it's happening sporadically across more than 1 application would make it appear to be a Chase issue and not an issue with the scripts themselves.

      -Kevin N.

      Delete
    3. It seems to be working now, some hours later. It's reasonable to assume that it was Chase, although it might be useful to know if Pocketsense was actually sending exactly the same thing and getting different results.

      Is there any easy way to capture the data Pocketsense sends to the financial institution?

      Harold

      Delete
    4. The Chase web site was down at the same time that the ofx.validation error was being returned. Safe to say it was a server-side issue

      Delete
  52. Harold, add the group of 5 # lines to ofx.py around line 332, and temporarily un-comment the last 4 of them (indentation doesn't copy correctly):

    SendRequest = True
    if Debug:
    print query
    print
    ask = raw_input('DEBUG: Send request to bank server (y/n)?').upper()
    if ask=='N': return False, ''

    #log rq file to xfr as *.ofxqqf (remember to remove files from xfr folder)
    # qqfFileName = ofxFileName + "qqf"
    # qqf = open(qqfFileName,'w')
    # qqf.write(query)
    # qqf.close()

    #do the deed
    client.doQuery(query, ofxFileName)
    if not client.status: return False, ''

    ReplyDelete
    Replies
    1. Thanks! Next time it happens, I'll try logging out and seeing if my PC is sending anything different!

      H

      Delete
  53. Something has changed at Chase since Sunday. Pocketsense does not appear to be affected. I have seen errors in other software (The error shows up as NOT IMPLIMENTED.), but this software continues to run fine. I have been using the combined ofx file and have not noticed anything out of the ordinary.

    Raymond

    ReplyDelete
  54. I used to download my Discover statements through Pocketsense. Then it stopped working in January as everyone here has noticed. Then last week Discover changed my CC account number. When I now try to download my transactions using the latest Pocketsense beta, the OFX file has an error that I'm not authorized to access the account and I need to register my account "through the online account center". I've already confirmed that I'm using the correct new CC, username, and password in Pocketsense.

    Discover helpdesk was useless because I'm not using Quicken. I did manage to find a registration page by digging into the discover.com front page, but when I register my card there, it tells me I'm already registered and offers to change my password. Does anyone know where I can register my new Discover CC for OFX access?

    ReplyDelete
    Replies
    1. Just to verify... you did delete your account entry in Setup and re-add it using the new card number?

      Delete
    2. Apologies for the slow reply. I had to delete and readd my Discover account in PocketSense. Simply modifying the existing account did not work.

      Delete
  55. Any luck with getting TD Bank to work with these changes? It requires the useragent to be set. Not sure if it worked with previous versions because I just got a TD Bank account.

    Thanks

    ReplyDelete
    Replies
    1. By useragent, are we talking about the http user-agent header field?

      Delete
    2. Yup. That's what I read elsewhere on this blog and in the moneysense forum.

      Delete
    3. Hmmm. I tested a the sites I use with and w/o a user-agent defined. All but Discover work with or without. The decision, then, is whether this should be a parameter that defaults to off or on. Since Discover is probably one of the most widely used, and most servers don't seem to care, I'm leaning towards a site option to enable the user-agent field, but default to none.

      Anyone have an opinion?

      Delete
    4. Scratch that. I'm just going to implement a retry w/ the user-agent enabled, if it fails without. It may be that a site requires a specific user-agent name. If so, I'll just add it as a site option.

      Delete
  56. We need to learn more about that field. Sounds like a good strategy Robert!

    Maybe that is a needed field for other sites that don't work with Pocketsense, like Merrill Lynch?

    ReplyDelete
  57. Any help you can give with getting Merrill Lynch to work is appreciated. Pardon my ignorance but what exactly is the user-agent field? Is it something specific and unique to each financial institution? If so, where might they be listed?

    Thanks.

    ReplyDelete
  58. I didn't know either, but apparently it is part of the request header and identifies the application requesting the information from the server, typically a browser, and helps the hosting server determine the best way to provide the requested information. Robert may have to use a dummy browser identifier??, since a Python script is actually making the request using the appID=QWIN, identifying the requesting App as Quicken.

    ReplyDelete
  59. Beta posted for testing. Implements a retry w/ user-agent if download fails without it.

    https://sites.google.com/site/pocketsense/ofxpy_pocketsense_beta.zip

    As for the user-agent header, it's a standard http field, and can be used to identify the client (e.g., Chrome, etc.). The beta simply uses "PocketSense". If that doens't work, and if someone has a specific value that's needed, then I'll need to revisit.

    ReplyDelete
  60. Your willingness to address these rare issues is really appreciated. And I can't believe how quickly you put out a beta. Unfortunately, I tried it with Merrill without success. Here's the OFX file it returned:
    OFXHEADER:100

    DATA:OFXSGML

    VERSION:103

    SECURITY:NONE

    ENCODING:USASCII

    CHARSET:1252

    COMPRESSION:NONE

    OLDFILEUID:NONE

    NEWFILEUID:0dca0c1c-f736-4b25-b985-7ffccd3e14d8



    :OFX::SIGNONMSGSRSV1::SONRS::STATUS::CODE:2000:SEVERITY:ERROR:MESSAGE:Your request cannot be completed at this time, for assistance please contact us at 1.800.MERRILL (637.7455).:/STATUS::DTSERVER:20170410183148.913[-4:EDT]:LANGUAGE:ENG:FI::ORG:Merrill Lynch & Co. Inc.:FID:5550:/FI::/SONRS::/SIGNONMSGSRSV1::/OFX:

    (In order to post the OFX file here I had to replace the left and right carat marks with : as the carats were being interpreted as HTML code).

    I'll be interested to see if anyone else has better luck.

    ReplyDelete
    Replies
    1. Have you tried ofxVer=102 ?

      Delete
    2. Just tried ofxVer=102. Same exact error message except it shows Version:102. Oh well :(

      Delete
    3. Curious about the next part, but have no way to test. Change line 296 in rLib.py to the following (ver 1.x header definition):

      SECURITY:TYPE1

      Delete
    4. I edited rLib.py as indicated, ran it, checked to be sure rLib.pyc showed an updated time stamp. Then tested the Merrill account and, interestingly, the OFX file still showed SECURITY:NONE. I'm pretty sure I did everything correctly. I'm guessing Merrill isn't looking at that setting and just returns a canned response.

      Delete
    5. Since I don't use Merril, I can't really test. Another quick check would be to see what response you get using OFX 2.x. I started adding support when debugging the Discover issue. Set ofxV=211. I doubt this is the problem, but easy to check.

      Delete
    6. Robert, thanks for your efforts. I changed ofxver from 102 to 211 in my sites.dat file. The test didn't generate an error, but when I imported the file in Money I got the message: The file you attempted to import appears to be invalid or contains corrupt data....... I took a look and all I got was the header info (see below); there was nothing else in the OFX file.

      OFXHEADER:100
      DATA:OFXSGML
      VERSION:102 (I know it says 102 but it's 211 in sites.dat for Merrill)
      SECURITY:TYPE1
      ENCODING:USASCII
      CHARSET:1252
      COMPRESSION:NONE
      OLDFILEUID:NONE
      NEWFILEUID:NONE

      I'm going to stick with just downloading the OFX file directly from the Merrill site. I don't have many transactions so don't need to do it very often. Thanks again for your help.

      Delete
    7. After posting I realized you may have meant to change Version in rLib.py so I changed that. Same issue as above, but header now says VERSION:211.

      Delete
  61. Wow. That's amazing how fast you got that up. Thanks a lot! And that did fix TD Bank!!! Much appreciated

    ReplyDelete
  62. And just to confirm, it works great with all my other accounts also!

    ReplyDelete