A new version of the scripts is available that uses the v10 Yahoo quote service. I did very limited testing, so comment if a symbol fails. Other than updating the script package, you will also need to comment (#) out the yahooURL option in sites.dat (i.e., do not use the custom yahooURL option... well, unless it matches the new default v10 service).
Download: ofxpy_pocketsense.zip [last updated 6/23/2024 10:30am EST]
Required: Disable (#comment out) the yahooURL option in sites.dat.
If you haven't updated in a while, be aware that the requests package is also required. See step #1 in the Install & Setup instructions for more info.
-Robert
UPDATES:
6-22-2023: Logging functionality added. A log file is now created for the GETDATA and SETUP scripts (i.e., getdata.log and setup.log). An important note is that enabling Debug (via control2.py) will write complete ofx messages to the associated log, including full request and response packets. This means that sensitive data typically found in ofx messages may be written to a log when Debug is enabled (e.g., username and/or password). A warning message will be shown when starting either script if an ofx username or password field is found in the log, as a reminder. It is recommended that log files be deleted after using Debug mode to start fresh with empty logs, since standard (non-debug) logging does not contain sensitive info. Needless to say, do NOT share log files without sanitizing.
Hmmm... It just opens and closes for me with errors:
ReplyDeleteC:\Users\TimPerkins\OneDrive\pocketsense\20230527>Getdata.py
Traceback (most recent call last):
File "C:\Users\TimPerkins\OneDrive\pocketsense\20230527\Getdata.py", line 56, in
import ofx, quotes, site_cfg, scrubber
File "C:\Users\TimPerkins\OneDrive\pocketsense\20230527\ofx.py", line 82, in
import requests, collections
ImportError: No module named requests
I don't have a requests file... maybe I'm running a really old version of PS?
FYI I'm running Python 2.7.17 (v2.7.17:c2f86d86e6, Oct 19 2019, 21:01:17)
DeleteSee the blog post on February 24, 2021 for instructions or Install & Setup step #1. The scripts have required the python requests package since then.
ReplyDeleteThanks, that worked! FYI the script is crashing if there's an old symbol in the sites file (FB in my case). Removing the old symbol fixed it and I'm back in business again! I did a copy\paste of the output (below) if you want to debug things:
DeletePocketSense OFX Download Python Scripts, Ver: 2023-May-26
Download transactions? (Y/N/I=Interactive) [Y]
Download interval= 7 days
No accounts have been configured. Run SETUP.PY to add accounts
Searching .\import\ for statements to import
Getting security and fund quotes...
Y: ^GSPC 4205.45 05/26/2023 14:20:03 1.30%
Y: ^IXIC 12975.69 05/26/2023 14:15:59 2.19%
Y: ^DJI 33093.34 05/26/2023 14:20:03 1.00%
Y: APA 33.41 05/26/2023 13:00:04 0.33%
Y: BAC 28.31 05/26/2023 13:00:01 0.50%
Y: BCS 7.81 05/26/2023 13:00:02 0.39%
Y: C 44.60 05/26/2023 13:00:46 0.84%
Y: DFS 102.36 05/26/2023 13:00:02 2.48%
Y: F 12.09 05/26/2023 13:00:14 6.24%
Traceback (most recent call last):
File "C:\Users\TimPerkins\OneDrive\pocketsense\Getdata.py", line 192, in
status, quoteFile1, quoteFile2, htmFileName = quotes.getQuotes()
File "C:\Users\TimPerkins\OneDrive\pocketsense\quotes.py", line 365, in getQuotes
sec.getQuote()
File "C:\Users\TimPerkins\OneDrive\pocketsense\quotes.py", line 124, in getQuote
self.getYahooQuote()
File "C:\Users\TimPerkins\OneDrive\pocketsense\quotes.py", line 163, in getYahooQuote
quote = pdata['quoteSummary']['result'][0]['price']
TypeError: 'NoneType' object has no attribute '__getitem__'
See post below for update re crashing. Just grab the latest version; same link above.
DeleteRealized after posting that I had forgot to re-enable exception handling for quote service, so just now refreshed the download zip.
ReplyDeleteThanks so much for your efforts, Robert. As I typically download 110 +/- quotes on a nightly basis, this is very handy. After your update, I only had one issue with a mutual fund quote for TSVPX which is a current fund so maybe just a hic-up.
DeleteJust did a quick test and see the issue w/ TSVPX. Will patch in next update (~ tomorrow likely).
DeleteJust tried it out using last Friday's closing prices. Working flawlessly, once again. Thank you for all you do with PS and hope to keep it going. Coffee is on me today! Try to enjoy the holiday weekend some.
DeleteIt seems that Ver: 2023-May-27 is downloading yesterday's closing quotes. Is that the intended behavior?
ReplyDeleteIt uses the regularMarketPrice field and associated time. For today, that would be COB yesterday for most, but during active trading it should be the current value. Of course, Money only records one entry per day.
DeleteHi Robert,
DeleteThank you for your reply and your continued support of PS.
Being retired, I sometimes lose track of the day. I didn't realize that today is Saturday so, of course. PS quoted yesterday's close. I'll see myself out.
@Kevin 👍
DeleteThank you for the update. Just tested it on 32-bit Miniconda. Everything worked fine for me.
ReplyDeleteYes, thanks Robert for the update and your continued support of PocketSense!!!
DeleteThanks so much, Robert. It worked perfectly.
ReplyDeleteThanks Robert for the update. In some of the quotes, shortName field is null (for example, SSAQX), it will cause exception and skip the quote (line 164 of quotes.py), you may want to add a check there.
ReplyDeleteI noticed it too via TSVPX fail (above). Fixed.
DeleteGood morning. ETrade, New Code - my taxable account continues to work as it has for a long time. For my retirement IRA I get the following error:
ReplyDelete** An ERROR occurred sending POST request to https://ofx.etrade.com/cgi-ofx/etradeofx
Exception type :
Exception val : 'ascii' codec can't encode character u'\xae' in position 5674: ordinal not in range(128)
HTTPS ResponseCode : 200
HTTPS ResponseReason: OK
Any ideas on what is happening? The previous code I was using (Ver: 04-Jan-2020) does work for this IRA.
That's a new one. I'm guessing the ETrade response has unicode. If so, a small change should fix. Likely this ~ evening.
DeleteThanks, appreciated as always
DeleteReady to test.
DeleteFixed, very cool, thank you!
ReplyDeleteGetting an error of writing quotes.xfr -- Will not update except for what is pre loaded. Any help greatly appreciated!!
ReplyDeleteSounds like the scripts are in a folder that doesn't have write access? Try putting it in \users\yourAccount\ofx, or something similar.
DeleteAfter activating Getdata.py, I wanted to change the default response to 'Download Transactions?' from (Y) to (I).
ReplyDeleteIn Getdata.py, I changed line 96 from ...Y/N/I=Interactive Y... to ...Y/N/I=Interactive I...
and changed line 97 from doit = Y... to doit = I...
These edits do change the default to (I) and it seems to work as expected.
Q. Are these the proper changes to make?
Q. Are there any other edits I should perform?
Q. Is there a better/different way to make this change of the default response?
Sounds reasonable. There's no setting to make it default, since it was mostly intended for debug.
DeleteThanks Robert. Appreciate it.
DeleteKevin N: There may be an alternative to modifying the python file. I usually invoke getdata.py via a batch file. Here is the line of the batch file:
Deletegetdata.py < lines.txt
If I entered that right, there will be a less-than sign between getdata.py and lines.txt.
The lines.txt, in my case, only contains blank lines. So for each question, getdata takes the default, as if I had pressed the enter key. In your case, I think you could put an I on the line. Experiment. An advantage of not modifying the *.py file is that you could update to a new version someday, without having to modify the future files.
Cal Learner.
@Cal Learner,
DeleteThank you for the suggested alternative to editing Getdata.py. It does work but, unfortunately, my experience with doing so doesn't allow me to actually be [ I ] Interactive.
I tried entering I, (new line) V, (new line) N (or Y) into Lines.txt and then running the batch file.
Getdata.py runs, gets quotes but then closes and sends the results to MS Money. (except when I entered N in the third line of Lines.txt)
Entering just I, or just I (new line) V, as expected, generates the same results.
For me, the desired result is for me to run Getdata.py, inspect the quote results and if necessary, X out of Getdata.py so as to 'not' send the results to MS Money. As you are aware MS Money only accepts one quote per day, so there I times when I don't want to send the results to Money.
That is as far as I got with it. Unless there is something else that you can think of to try, I will continue with editing Getdata.py so that [ I ] = Interactive is the default.
Newer PocketSense scrubber posted. More sophisticated Citi bank handling and more.
DeleteI made a newer version of the older scrub_cl4a.py. It is available as
https://drive.google.com/file/d/1vQVSzcjOQ9gOjHoZTAxxTltceM9YKRBu/view?usp=sharing
File cl6_0.zip has two files in it: scrub__CL6.py and CL6note.txt
More details in the notes file, but it enhances the Citibank web-downloaded scrubbing.
It also incorporates some Fidelity changes. It also allows running the scrub_cl4a.py as a program for some separate testing.
Remember to rename scrub_cl4a.py to old_scrub_cl4a.py or delete the old, to prevent the older file from also running.
Questions after you have taken a look? Glad to address those.
Cal Learner
Strange, but my ofx citibank download doesn't contain anything with "citi". It does have a BANKID that I can test for. "073456789"
DeleteYou are correct. I need to make a correction. Not that a QFX file does have "Citibank" in it at about line 21. But the OFX files and the QFX files both have <BANKID>073456789
DeleteCal Learner
Fixed that Citibank error for files put into the import folder. CL6_1.zip is at https://drive.google.com/file/d/1VJPPkb_XgxBg-40QklPnjvKHwGAD_NBp/view?usp=sharing
DeleteCal Learner
Should I still be running only version 2.7 of python to make this script work? Thanks for the quotes update
ReplyDeleteWhen I try to add (my Fidelity) accounts in the setup.py file (after updating sites.dat), the program just disappears from my screen after I put in my username and password and hit enter!
ReplyDeletesorry --- ignore
Deleteok.. thanks ..didnt know that - default is "anonymous" and I dint think to change it
ReplyDeleteHi Robert,
ReplyDelete>May 28, 2023 at 11:42 AM That's a new one. I'm guessing the ETrade response has unicode. If so, a small change should fix.
Q. Is it possible that the above fix could break logging into a bank and/or credit card account?
Using Ver: 2023-May-28, My connection to Fidelity Investments works as expected. However, my connection to Regions Bank & Regions Credit Card now throws the following error 'The user cannot signon because he or she entered an invalid user ID or password.'
Using both Ver: 04-Jan-2020 and Ver: 07-Mar-2020 BETA, the connections to Regions work as expected.
@Kevin: Is this the first time you're using a version that uses the requests pkg? I ask because those versions used the older http library, which broke w/ some sites. I assumed everyone migrated to the new since then, so that bugs had been worked out, but maybe not.
Delete@Robert,
DeleteI have Active State 2.7.18 installed.
C:\>pip install requests
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Requirement already satisfied: requests in c:\python27\lib\site-packages (2.23.0)
Requirement already satisfied: chardet<4,>=3.0.2 in c:\python27\lib\site-packages (from requests) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in c:\python27\lib\site-packages (from requests) (2.9)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\python27\lib\site-packages (from requests) (1.25.8)
Requirement already satisfied: certifi>=2017.4.17 in c:\python27\lib\site-packages (from requests) (2019.11.28)
WARNING: You are using pip version 19.3; however, version 20.3.4 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
C:\>
I use one instance of PS for quotes (currently, Ver: 2023-May-28) and a second instance for transactional downloads (currently Ver: 07-Mar-2020 BETA)
I should add that I've been using Active State from day one.
DeleteDo you think that upgrading the pip version per the above output would have any effect?
Try adding "userAgent: none" to the site entry in sites.dat.
DeleteHi Robert,
DeleteAdding 'userAgent : none' to their respective 'site' entries in sites.dat has resolved the issue for both Regions Bank and Regions Credit Card.
Well done and THANK YOU!
Sorry it doesn't allow using google account even though I removed blockage of 3rd party cookies. My problem with the whole setup.py disappearing of the screen when I try to enter account data still exists. It must be something wrong with my installation (ver 2.7 python with latest ofx_pocketsense.zip)...any ideas?
ReplyDeleteignore this, please - i am just getting very ancient...thanks for all the hard work
DeleteNew PocketSense works fine for new quotes url, however, downloads from Fidelity Net Benefits fails. Fidelity Investments are okay. The previous version works for Fidelity, but fails for quotes.
ReplyDeleteI am using ActiveState version 2.7.18 and have upgraded pip requests to latest available.
Ray
Thanks Ray. To clarify, when you say "previous version", you mean the most recent prior to the quotes update... correct? Meaning, you have been using a version that uses *requests*.
DeleteMy bad on the about the previous version. It is "20-Nov-2019".
DeleteThis is the error message:
HTML HEAD
TITLE Access Denied /TITLE
/HEAD BODY
H1 Access Denied /H1
You don't have permission to access "http://nbofx.fidelity.com/netbenefits/ofx/download" on this server.
The url in sites.dat is " url : https://nbofx.fidelity.com/netbenefits/ofx/download
The Fidelity Investment account uses a different url and it works fine.
Ray
Forget the previous message....httpss gave a different error message....its not http vs https???????
ReplyDeleteRay
FOUND IT!!
DeleteIt has to do with the userAgent. I will fix mine and try it out later today. If I blank out the userAgent in file ofx.py line number 273 ... Fidelity net benefits does not error out ...I will fix it properly later by using sites.dat and test when I get a chance.
Sorry to have bugged you with this minor inconvenience.
Ray
Good to know. Add "userAgent: none" to the site entry in sites.dat.
DeleteThanks Ray! I added that to my OFX Download Settings page:
Deletehttps://microsoftmoneyoffline.wordpress.com/2010/10/06/cal-learners-review-fidelity-401k-citi-card-and-vanguard-account-info/
@NexTek, Regions Bank also requires the "userAgent : none" entry in sites.dat
DeleteGot it. Thanks.
DeleteI'm getting this error when testing my stock quotes via Setup.py. Any ideas?
ReplyDeleteTraceback (most recent call last):
File "Setup.py", line 396, in
test_quotes()
File "Setup.py", line 211, in test_quotes
status, ofxFile1, ofxFile2, htmFile = quotes.getQuotes()
File "C:\Users\cddil\OneDrive\Documents\Financial\MS Money\Money Work Around\ofxpy_pocketsense\quotes.py", line 391, in getQuotes
ofxFile2 = xfrdir + "quotes" + dateTimeStr + str(random.randrange(1e5,1e6)) + ".ofx"
TypeError: cannot concatenate 'str' and 'function' objects
Bug/fixed. You're the first to use the latest w/ the forceQuotes option enabled :|
Deletep.s. That option was for non-US version of MS Money and should be disabled for US versions or other purposes.
DeleteThanks! And maybe sorry? :) I'll try things with forceQuotes off. I'm using MS Money 2000 (and not the Sunset edition you mention elsewhere) so not sure if that's possibly a source of the issue.
DeleteHi Robert. I appreciate what you do to help all of us. I have used ofxpypocketsense for a couple of years. However, when I tried to use this updated one by double clicking Setup.py, a window appear and disappear within a second. I tried to re-install the Python many times but do not understand how to type "C:>pip install requests" in a command prompt. Can you or anyone please guide me on how to install and run it again? I think I need step-by-step guide.
ReplyDeleteI just did this a few minutes ago. I had to launch python and then type "pip install requests" in the command prompt that Python gave me. So, at the C:\> prompt, I typed "python". I got some information displayed showing that I was now in the Python application. Then I typed "pip install requests" and it spit out a bunch of status messages showing it was happening. Then you can exit Python and exit your command prompt window and should be good to go.
DeleteThank you cddiller. But I don't even know how to get to C:\> prompt. What I got is C:\Users\my name>. And if I type "python", it just send me to a new window with search result for python. I really feel like an idiot.
DeleteIf you are using Windows PC, in search box type cmd --> lunch command prompt appl. Normaly it will start with C:\Users\your_name> Type cd\ --> enter. after that try: pip install requests
DeleteOk. I fixed it. In case someone can be as clueless as me, first I need to select all the options for python to include "PATH" when install the program. Then I just type "pip install requests" at the C:\> prompt.
DeleteThank you cddiller and Arkady for helping me. I might have given up already without you two.
Is anyone else having issues being able to pull OFX data from USAA by chance? I believe it worked one time with the new Pocketsense script this morning but since then, it's producing some kind of OFX error.
ReplyDeleteI have been using ofxpypocketsense to download stock quotes only to MS Money. I do not have any bank or credit card accounts in the US so I have never add any accounts and just skip that part in sites.dat. Today after update the ofxpypocketsense, I use Getdata to download quotes but it shows this:
ReplyDeleteDownload transactions? (Y/N/I=Interactive) [Y] y
Download interval= 7 days
No accounts have been configured. Run SETUP.PY to add accounts
Searching .\import\ for statements to import
Press to continue...
I can't get the quotes as before. Is there a way to download the quotes?
Run setup.py (python setup.py) and enable stock quotes.
DeleteIt works now. Thank you Robert.
DeleteMay I also ask why sometime the date of the stock quotation is wrong? Is it becuase of yahoo site?
DeleteThis comment has been removed by the author.
DeleteJust a quick update...It seems that when I use the new version of Pocketsense, it stops working with USAA. Robert, can you confirm if those modifications made in the DEV version for the USAA fix were brought into this version?
ReplyDeleteI don't know if this will help with USAA today. I have no means of testing that myself.
DeleteThe original method of scrubbing, I think, was to modify the scrubber.py file for those FIs that needed some scrubbing action. This modification was done by Robert for some, but I had done my own modifications, particularly for Fidelity.
I don't use USAA, but in March 2021 I made a scrubber based on items suggested by Evan April 2, 2021 and info from Ron(remove transactions if the FITID starts with MD, per Ron).
Starting some time in 2021 there was a new method of adding scrubber action for particular financial institutions. Any file with the name of the form scrub_*.py was treated as a special scrubber, and that file needs to only operate for those FIs that it is intended for. This does not replace the scrubber.py file that comes with each PocketSense distribution.
My latest iteration includes scrubbers for Fidelity, Janus, Vanguard, USAA, Citicard, Citibank(web), PNC(via cfree) .
I put these all in one file as a convenience to me, but I had thought of turning these into several one-FI scrub_*.py. For now I have been issuing these together. The latest version is scrub_CL6.py, and is available as a zip file at
https://drive.google.com/file/d/1vQVSzcjOQ9gOjHoZTAxxTltceM9YKRBu/view?usp=sharing
File cl6_0.zip has two files in it: scrub__CL6.py and CL6note.txt. The note points out that if you used one of the earlier scrub_*.py files, you should rename or delete the older one.
Cal Learner
Update worked flawlessly. Coffee donation on its way... the amount of hours this software has saved me is incredible. First entry 6/16/1999 - almost 24 years of all fiscal data in one place. Microsoft's best piece of software ever!
ReplyDeleteApart from finally being compelled to upgrade my Python install, flawless. Thanks for all you do Robert! A few coffees on me
ReplyDeleteCan anyone else who uses USAA confirm whether the script fails after a period of time? Or is it just me? I keep having to go to the USAA OFX accept site and click the accept button to get the script to download ofx statements every so often. This didn't happen with the previous DEV version that was released to fix the USAA download issue when they switched to having to use the clientUID and PIN.
ReplyDeleteYes, same here.
DeleteI had to re-auth with the DEV version not too long ago (See some of the threads in the previous post, http://pocketsense.blogspot.com/2022/11/at-crossroad.html?showComment=1684766086989#c7813174158945277591). I'm wondering if USAA changed something on their end to require more frequent authentication.
DeleteWhen changing PocketSense versions, I always copy the old connect.key and ofx_config.cfg and sites.dat from the old configuration. You don't want a new PocketSense version generating a new connect key, which is what I think would happen if you did not copy your existing connect.key file.
DeleteI am not saying that is what caused a need for re-auth for USAA. If that did not happen when you went to a new PocketSense version or to a new computer, then my files thoughts were not relevant.
Cal Learner
THANK YOU for updating!! (I only use this to get stock & fund quotes, not to download any actual account statements. I had done some modifications to original pkg & I'm having trouble getting things to work now without any real account/statement.... I think I can get it figured out, but would be nice to have that option built-in to the scripts...
ReplyDeleteCitibank banking from import
ReplyDeleteCitibank Credit Cards work well with pocketsense, but the banking accounts are not supported with a pull. I can download the OFX file after logon in the user interface, but the DTEND file is not updated.
...
20230603
20170204210000
...
Can a scrubber_citi.py be added to change the DTEND to now? I saw the the "import" directory is now active. This date is published as "Last Updated" on the Money Account List screen.
Tom: in my scrub_cl5A.py which is in in scrub_cl5A.zip, I have a citibank scrubber that defines and calls
ReplyDeletefunction check_order_DTSTART_DTEND(ofx): #insures DTEND is after DTSTART
This function may fix the problem for you.
See post in https://pocketsense.blogspot.com/2021/08/dev-testing-continued.html#comment-form by Cal Learner September 12, 2021
Cal Learner
Thanks Cal. You're code was great. Thanks for sharing. I'm neither a python coder nor have an environment to develop/test, but I now have scrub_citibank that works with extracts from your share. I was wondering if there would be a way to run the imports from setup.py.
DeleteTom: I tried downloading, and putting the file(s) into the import folder. Then I tried some things in setup.py, but nothing happened.
DeleteMaybe you can help me. The interface to get the qfx/ofx files from the Citibank website has changed several times over the years. Include in that were times that you could not get all accounts at once, and had to get things separately. Now the navigation is weird, but I can deal with that.
There have been times when the FITID for the same transaction would be different each day I downloaded. The workaround was to specify a download date range that did not overlap with previous downloads. Now I cannot figure how to get all accounts at once. Also, I don't know how to specify a date range.
Any suggestions or info on that?
Cal Learner
Money didn't import my SAVINGS "import" from Citibank until I commented out the timezone fix. I' in the Central US timezone.
Deletedef check_order_DTSTART_DTEND(ofx): #insures DTEND is after DTSTART
dtstart_date = re.search('.*\s*(\d\d\d\d\d\d\d\d).*', ofx).group(1) #get yyyymmdd only
dtend_date = re.search('.*\s*(\d\d\d\d\d\d\d\d).*', ofx).group(1)
if (dtstart_date > dtend_date): # that would be abnormal so swap dates in thatcase
scrubPrint ("swapping dates for DTSTART and DTEND")
dtend_old=dtend_date
dtend_date=dtstart_date
dtstart_date=dtend_old
# dtstart_date=dtstart_date+"1200" #avoid time zone problem
# dtend_date=dtend_date+"1200" #avoid time zone problem
ofx= re.sub( "\s*\S+[^\s<]",""+dtstart_date , ofx)
ofx= re.sub( "\s*\S+[^\s<]",""+dtend_date , ofx)
return ofx #as modified string to be used as date
Thanks for the Yahoo quote fix! I've Paypal'd a few bucks to your coffee account in appreciation
ReplyDeleteJust dropping a note to thank you for maintaining and providing these scripts for everyone. I've been using Money since 1998 and your scripts since 2011, really happy to see it continue for so long. I sent some coffee your way as well.
ReplyDeleteI've been away on family vacation since posting the last update, so thanks to everyone for the feedback and coffee tips :) Regarding the USAA issue, it's unclear what may be happening there. I'm going to add logging support to help with such issues, but it will be mostly for future issues since the scripts currently only have crude debug/log functionality. It's something that should have been added long ago, but somehow never happened.
ReplyDeleteI'll post another thread when ready.
Hi Robert, I had an update with regards to my USAA account. I myself noticed that, after a period of time, I stopped having to go to the USAA OFX site to reauthorize the script so now, it seems to be working. It's been good for about 2 weeks now. Thank you again for your help!
ReplyDeleteI'm still having no luck downloading USAA accounts, both banking and credit card. I've reauthorized "Quicken" on the USAA site several times, used the provided ACCESS ID and ACCESS PIN in PocketSense setup, but I keep getting "An error occurred requesting accounts from the site. Please check username and password." Thanks!
ReplyDeleteI don't use USAA, so can't help much w/ setup via their site. Hopefully another user can chime in and help. Are you saying that it worked before, but not now... or are you setting up for the first time?
DeleteMy limited input is to review site examples for USAA. All of the fields shown are required, I believe, including clientUID. The "Your_ClientId_From_The_Allow_Page" key value is found somewhere on the USAA authorization page. I don't know where exactly, but recall a mention that it is embedded in the browser link itself?
USAA downloads had been working for me ever since they implemented the Quicken authorization process, but as of May 26, downloads have stopped working.
DeleteI expect "appver : 2300" should be changed
Deletefor USAA in https://microsoftmoneyoffline.wordpress.com/2010/10/06/cal-learners-review-fidelity-401k-citi-card-and-vanguard-account-info/
I expect "appver : " would be good, since that causes that to come from control2.py of the PocketSense release that you are using. I don't use USAA either.
Cal Learner.
New version posted to include logging functionality. Note added to this thread post (above).
Delete@Cal regarding logging from custom "scrublet" modules:
DeletescrubPrint('message') now logs (info) messages to both screen and file.
If more explicit logging control is desired, import the logging module and create a 'root' logging object in the module, e.g.:
import logging
log = logging.getLogger('root')
It's then simply log.debug('message'), log.info('message'), etc up to log.exception('message'); where log.exception() includes traceback.
The config file for MoneyDance (https://moneydance.com/synch/moneydance/fi2004.dict) says to use appver 2400
ReplyDeleteThank you Cal + Anonymous. I removed appver + appid from USAA Banking, as well as USAA Credit Card settings. I suspect both 2300 and 2400 were from days long past when the default version was much older (1800 or so). If anyone finds settings that differ from that work better, I'll be glad to alter once again.
ReplyDeletelast comment was by ameridan, not another anoymous.
Delete@Robert, thank you. Sadly, I'm not very competent in Python, so I'm not sure if I enabled debug correctly: I edited control.py and moved the # from the True line to the False line; is that right? Tried adding my USAA acct, got the fail message, but I don't see any detailed logs in xfr, only the original-style log.
ReplyDelete@ameridan, I removed appver and appid (#'d them out), but result was same as before.
getdata.log and setup.log will be in the main folder (w/ the scripts themselves), not in xfr.
DeleteAlso... logging was added as-of yesterday, so be sure you're using the latest.
DeleteGotcha, I found the setup.log. Tried adding USAA account, but setup window closed immediately after entering the Acct Password, no warning/error messages--it just closed. The log did capture my inputs and this: "ERROR - An error occurred when executing query."
ReplyDeleteThe only debug.log is dated Dec 2020 and contains no recent info.
Re: setup.log: Does it not contain a traceback message for the error (immediately after) ?
DeleteI tried to force a crash in the setup module and saw a bug. Not related to usaa failing, but was related to not capturing full debug log. Considering logging is new, I won't be surprised to find other bugs missed when implementing :|
Deletep.s. Update posted.
DeleteLatest version (updated 6/23/2023 2:10pm EST) seemed to work this time. There is a whole bunch of additional debug info in setup.log and getdata.log now. Do you have time to look at the output? And would you want the entire (sanitized) log files or extracts? Thanks for your time!
ReplyDeleteI can look at basic script-related debug info, but not much related to USAA itself, as I don't use them.
DeleteBefore doing much, what version of Python are you using? If unsure, open a cmd prompt and enter "python". Version info will show at startup.
I'm on Python 2.7.18
DeleteI don't mind looking over a *sanitized* debug session log to see if anything jumps out. Delete the current logs, run Setup, and attempt to add the account. When sanitizing, replace sensitive info w/ useful phrases like "myUserName", "myPassword", "myAccount#", etc. Send to pocketsense1/at/gmail/dot/com
DeleteAlso, a copy of your sites.dat entry for USAA will help.
DeleteI'm getting an error with this new version. Here is the trace: Getdata.py Line 195... quotes.py line 370.... Line 120 in quotes.py. The call that fails is getQuote log.info('Getting quote for: %s' % self.ticker). NameError: global name 'log' is not defined
ReplyDeleteIgnore last comment. I apparently introduced a different bug yesterday. Stay tuned...
DeleteBug fixed (I hope). New version posted.
DeleteWorks now. Thanks.
ReplyDeleteUSAA users, if you have to go to the USAA site to re-enable Quicken, do you get the same Access ID and PIN each time? I do.
ReplyDeleteI'm using these values in my sites.dat file:
fid : 24591
url : https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
Tech support on USAA was unhelpful. They wanted me to open Quicken and I was unable to get across that I use a different program which uses the same access protocols as Quicken. Their database (unsurprisingly) doesn't include PocketSense, and they said I need to contact the 3rd party software company for help. The only useful information I could get from them was that Access ID and Access PIN never change, and Quicken no longer uses Access ID and Access PIN.
DeleteI use USAA for banking (and a credit card). Here's my sites entry:
DeleteSiteName: USAA Banking
url: https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
fiorg: USAA Federal Savings Bank
fid: 67811
bankid: 314074269
AcctType : BASTMT #bank
appid: QMOFX
appver: 2300
ofxVer: 103
dtacctup: 19900101
userAgent: InetClntApp/3.0
clientUID: ***the UID in the URL of the page that gives you your ID and PIN***
I think that the UID might have changed recently (a month ago?) but once I updated that I haven't had a problem.
Thanks, that info matches my current sites file. My clientUID hasn't changed since USAA implemented this process, based on my notes from Apr 2021. The Quicken permissions will disappear from USAA within a day or two of enabling them...I just re-enabled them again now, received the same Access ID and PIN, typed them in on the PocketSense Setup window, and received "An error occurred requesting accounts from the site. Please check username and password."
DeleteAndrew Shafer, just to confirm I'm still doing this right...are these the steps you followed to re-authorize OFX for USAA?
Delete1. Login to USAA
2. Go to https://www.usaa.com/accessid
3. Before clicking "Allow", record the client_UID from the URL on this page.
thanks!
I downloaded the version dated 30 May 2023 and it has been working fine for a few weeks (Vanguard and Fidelity only). An old 401k I have has moved custodian from Vanguard to an outfit called Voya so now I am trying to set that up. I got the following info from ofxhome.com and moneydance respectively:
ReplyDeleteFI Id 1289
FI Urlhttps://ofx.voyaplans.com/eofx/Server
sorry, errant key stroke...continuing...
DeleteWhat I got from ofxhome:
FI Id 1289
FI Url https://ofx.voyaplans.com/eofx/Server
What I got from moneydance:
{
"id" = "md:2543"
"uses_fi_tag" = "y"
"access_type" = "OFX"
"fi_name" = "Voya Institutional Plan Services"
"bootstrap_url" = "https://ofx.voyaplans.com/eofx/Server"
"fi_id" = "1289"
"app_id" = "QWIN"
"fi_org" = "ing-usa.com"
"app_ver" = "2400"
}
I'm not sure what to do with "id", "access type" . The below doesn't work, fails setup with "An error occurred requesting accounts from the site. Please check username and password." after entering same in step to configure Voya account.
This is what my sites.dat entry is:
SiteName : VOYA
AcctType : INVSTMT
fiorg : ing-usa.com
fid : 1289
url : https://ofx.voyaplans.com/eofx/Server
bankid : md:2543
brokerid :
ofxVer :
appid : QWIN
appver : 2400
mininterval: 10
timeOffset :
Anyone else use Voya? It's brand new custodian for this employer and just registered etc. today, but logged in and got all account balances and generated statement from website a couple hours ago.
Thanks very much,
Jon
I forgot to say I did it without the "md:2543 in "bankid" also with same result.
DeleteJoryde: Look at the *.ofx file in the xfr folder for clues. You might use Notepad to look at the file.
DeleteI would, in general change, "appver : 2400" to "appver :". That would cause you to use the 2700 version from control2.py. I know nothing about VOYA downloading.
Cal Learner
Thanks Cal,
DeleteI finally found the right combination! yea! So, for any others who may get their assets transferred to VOYA, here's what worked:
Sites.dat looks like:
SiteName : VOYA
AcctType : INVSTMT
fiorg : ing-usa.com
fid : 1289
url : https://ofx.voyaplans.com/eofx/server
bankid : #2543
brokerid :
ofxVer :
appid :
appver :
mininterval: 30
timeOffset :
The important detail is the user name must be your website login "username + @yourcompany", so in my case username@nissan (the website I use for regular access is nissan.voya.com). I also could not find my new account number anywhere on their site but pocketsense sent back a number@number, turns out it's the last 4 of your SSN @ the plan ID no. Anyway, it works. Still trying to figure out the best way to transfer the accounts in Money; the data obtained from VOYA adds the plan ID no. as the ticker so it seems all the investments have to be newly named. Small detail to be worked out to be sure.
I was on the phone with VOYA help for 45 mins (on hold for most of it); she tried real hard to find a supervisor that could help but they wanted me to contact microsoft!, ha! Said they only support Quicken and Mint. In the end I got it working during one of the hold spells and relayed to her the info so maybe they'll have it ready for the next poor sot that calls in.
Cheers,
Jon
Joryde, thank you for Voya instructions! Worked for me.
DeleteHi. I'm in UK so dont download any transactions and just manually manage them, but been using your great script for years to maintain prices using Yahoo. I downloaded the script and extracted the new files into the existing directory to overwrite the old files. I'm not sure if i need to do anything else, but double clicking setup or getdata both flash the screen, but no window appears? Thanks for any advice
ReplyDeleteAdding to Cal... see https://sites.google.com/site/pocketsense/home/msmoneyfixp1/p2 , step #1 for info about installing the *Requests* package.
DeleteI wanted to share an updated workaround to get CSV files into Money. Like some of you, I had used the XL2QIF macro for a long time, but it wouldn't work on 64-bit Excel. Recently, someone modified the macro script to be 64-bit compliant, and we can once again convert a CSV to QIF.
ReplyDeleteHere's to YT video explaining the original macro:
https://www.youtube.com/watch?v=B4aL7WYugC8
The last video comment by @DavidRosengarden explains how he solved the issue and uploaded the repaired files to dropbox (virus scan to be safe):
https://www.dropbox.com/s/exj8q0w7wci7kxe/xl2qif_1.11.complie.fix.zip?dl=0
Hi, I want to say thank you for continue to provide solutions on stock quote.
ReplyDeleteI recently notice that May 28,2023 version quote will put quote one day ahead.
For example, I clicked "getdata" to pull today's quote(July 12, 2023). On the python script, it shows correct quote and correct date. However, when I go to account summary, and it show the 7/11/2023 close price.
As an example, using SSO ticker:
If I click update price mannually,
It only have 7/11/2023 and 7/13/2023 price no 7/12/2023 price
It lists SSO on July 13, 2023 close at 57.86 (which is actually 7/12/2023) closing price.
Any idea where I did wrong? I run the python script first before I open the MSFT Money/sunset
Thank you.
Chip
Money only accepts a single quote per day, so the scripts use the "last close" price.
DeleteIs SSO a stock or a fund? I've never seen funds updated real-time in the US markets. I think funds are combinations of stocks, so the value is not available for download real-time. Just guessing...
ReplyDeleteTom: SSO is an ETF (Exchange-Traded Fund). Prices are updated thru the day like a stock. I always tell Money they are a stock. In the OFX files from Fidelity, they come in as a stock (such as being listed in a POSSTOCK record.
DeleteCal Learner.
Mutual fund prices, also known as net asset value (NAV), are updated once a day after the U.S. stock market close, usually between 4 p.m. and 6 p.m. EST.
DeleteClosed-end funds, however, don't have to update their price or NAV daily.
The new script seems to have stopped working for me today. It was OK a couple of days ago. Has Yahoo changed something again?
ReplyDeleteI just get "invalid quote response. Skipping..." for every request.
ReplyDeleteJust tested and worked for me.
DeleteNope, not working for me - "invalid quote response. Skipping..." for every quote request.
ReplyDeleteI am using "PocketSense OFX Download Python Scripts, Ver: 2023-May-28' - should i update ?
ReplyDeleteYes. There were some bug fixes after that, plus full logging.
DeleteWill try , but Robert , do you think it could be the reason why number of ppl getting "invalid .." or it something else ?
DeleteBTW, yesterday it was working without any problems ..
DeleteTried the June 2023 zip file, i am getting "invalid quote response", did yahoo change the server again?
DeleteTried the June 2023 zip file, I'm getting "invalid quote response". Any ideas?
ReplyDeleteSame here, "invalid quote response. Skipping." It worked fine yesterday, and days before.
ReplyDeleteGetting "Invalid quote response. Skipping." Was working until today July 13, 2023
ReplyDeletehttps://github.com/dpguthrie/yahooquery/issues - probably related
ReplyDeletehttps://github.com/dpguthrie/yahooquery/issues/203
Deletehttps://query2.finance.yahoo.com/v10/finance/quoteSummary/AAPL?modules=price
DeleteReturns: error.code: "Unauthorized", error.description: "Invalid crumb"
https://query1.finance.yahoo.com/v1/test/getcrumb returns vuD1doC400c, then
Deletehttps://query2.finance.yahoo.com/v10/finance/quoteSummary/AAPL?modules=price&crumb=vuD1doC400c
returns the quote
using the 'getcrumb' request from a browser works, but calling it from python 2.7 results in "Forbidden", exact same URL. I see that a company called 'https://marketstack.com/' provides 100 monthly requests for free which may satisfy weekly updates for a limited number of securities.
ReplyDeleteAlternatively, more stable interface to historic prices from Yahoo, returning CSV file can be used:
Deletehttps://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1689033600&period2=1689379200&interval=1d&events=history
One downside is that "today" closing price is no available until the market close for stocks and even later for mutual funds (NAV recalculation @~8PM PST)
Good morning, I'm getting these errors today.... Any idea what's going on?
ReplyDeletePocketSense OFX Download Python Scripts, Ver: 2023-May-26
Download transactions? (Y/N/I=Interactive) [Y] I
Download interval= 7 days
No accounts have been configured. Run SETUP.PY to add accounts
Searching .\import\ for statements to import
Getting security and fund quotes...
Traceback (most recent call last):
File "C:\Users\TimPerkins\OneDrive\pocketsense\Getdata.py", line 192, in
status, quoteFile1, quoteFile2, htmFileName = quotes.getQuotes()
File "C:\Users\TimPerkins\OneDrive\pocketsense\quotes.py", line 365, in getQuotes
sec.getQuote()
File "C:\Users\TimPerkins\OneDrive\pocketsense\quotes.py", line 124, in getQuote
self.getYahooQuote()
File "C:\Users\TimPerkins\OneDrive\pocketsense\quotes.py", line 163, in getYahooQuote
quote = pdata['quoteSummary']['result'][0]['price']
KeyError: 'quoteSummary'
All: I'm aware of the Yahoo Finance issue and will be looking at it. Stay tuned...
ReplyDeleteThank you Robert. For those in a hurry, as a quick temporary workaround, v6 of Yahoo API works. Add this to your sites.dat:
ReplyDeleteYahooURL: https://query2.finance.yahoo.com/v6/finance/quoteSummary/%s?modules=price
Another temporary fix is to change the value of self.YahooURL in site_cfg.py to
ReplyDeletehttps://query2.finance.yahoo.com/v6/finance/quoteSummary/%s?modules=price
Indeed it does! Now if only Yahoo would stop changing their working directory every few weeks and we could count on some stability..... Thanks for posting.
DeleteInteresting that the v6 service is still working w/o session credentials. Thank you, and I'll add it to the post above for now.
ReplyDeleteLast comment on this thread. New post for new version.
ReplyDelete