Tuesday, August 1, 2017

Patch / Update Available

(comments closed)

Last Updated: 8/27/2017

Recent changes at Vanguard prompted another round of minor patches to the scripts, which can be downloaded @ https://sites.google.com/site/pocketsense/home/msmoneyfixp1/p2


Detail: Vanguard is now using session tokens (cookies).  There were a number of ways to handle it, but I chose to allow the first connection attempt to fail, and return the server-response cookie on the next attempt, thus using the same tokens as the server.  We could instead send a client token on the first pass, and skip the double-connect, but that requires an assumption re: expected server tokens.  The hope is that the chosen method will support changes at other banks, should they choose to use session keys, regardless of token schema.


100 comments:

  1. Thank you again, update works!

    Kale

    ReplyDelete
  2. Setting a timeOffset in sites.dat, such as
    timeOffset : -30

    gives an error "type object 'datetime.datetime' has no attribute 'timedelta'"

    A cure is to change "from datetime import datetime" to "from datetime import datetime, timedelta" and to change "datetime.timedelta(hours=h)" to "timedelta(hours=h)".

    I don't know why that is needed.

    CL





    ReplyDelete
    Replies
    1. What version of Python are you using?

      Delete
    2. Never mind... it's a bug introduced somewhere along the way. I replaced datetime.datetime calls w/ a datetime import. Missed the timedelta part, since I don't use it. Will fix.

      Delete
  3. Thanks for all the work you have done.
    There is an issue i am facing. I am using Google finance and not yahoo finance. If i check for quotes for symbol "BOM:500012", the result given is wrong i.e. i think it is multiplying the code itself i.e. 500012 * the day's price and giving wrong result. e.g. today's rate is 32.05 but getdata is returning like G: BOM:500012 16025384.6 8/24/2017 4:00pm

    ReplyDelete
    Replies
    1. I see the problem, and you're correct re: the multiplier. The scripts allow a multiplier option for each quote line, and is specified using a M:# parameter. The BOM: is being interpreted incorrectly, so I'll modify the script to handle it better.

      Delete
    2. Thanks. The update works fine now.

      Delete
  4. The scripts have been updated to address the two issues mentioned above.

    ReplyDelete
  5. I finally got my acct defined properly with USAA, at least it said it downloaded successfully. But I get this error message:OFXAdapter: Failed to parse request: Unable to parse a composite field "OFXRequest.OFXRequest": postTag "/OFX" not found. I can see no OFX data when I open the XFER file other than that msg. Any idea where I go from here?

    ReplyDelete
    Replies
    1. Just a guess here, but try explicitly setting ofxVer:102 in sites.dat for USAA. If that doesn't work, see if they support ofx 2.x (e.g., ofxVer:211).

      Delete
  6. No, that did not work. Msg: OFXAdapter: Failed to parse request: Unable to parse a composite field "OFXRequest.OFXRequest": postTag "/OFX" not found.
    I will ask them if they support 2.X

    ReplyDelete
    Replies
    1. I did a quick search and found another comment from long ago: "Apparently, with USAA, leading 0's in an account number are really necessary". Just a fyi, since many banks/sites require leading zeros be included w/ the account number (e.g., 00001987567).

      Delete
  7. Yes, I already knew that from my past experience with Quicken. I use two leading zero's. I also found that if I did not get the acct number correct, the error msg tells me that. But in this case, it says the data downloaded, even though I get the error msg above.

    ReplyDelete
  8. I have multiple accounts at USAA that are working, here's the USAA section from my sites.dat:

    SiteName : USAA
    AcctType : BASTMT #bank
    fiorg : USAA
    fid : 24591
    url : https://service2.usaa.com/ofx/OFXServlet
    bankid : 314074269
    brokerid :
    appid : QWIN
    appver : 1700
    mininterval:
    timeOffset :

    I also noted that usernames must be zero-padded to exactly 9 digits and account numbers must be zero-padded to exactly 10 digits.

    Hope that helps!

    ReplyDelete
  9. Thank you. That is for USAA Federal Savings Bank, I am trying to get it to work for USAA Investment Management Co. which uses different stuff. I did know that the user ID needed to be 9 digits with leading zeros. I wonder if I can dummy in the appid and appver?

    ReplyDelete


  10. LarryB, what is your sites.dat entry for USAA Investment Management Co?

    CL

    ReplyDelete
    Replies
    1. Name: USAA Investment Mgmt Co
      AcctType: INVSTMT
      fi: USAA
      Fid: 24592
      url: https://service2.usaa.com/ofx/OFXServlet
      brokerid USAA.com

      Delete
    2. Is that a copy/paste from the actual sites.dat file? Some field names aren't what's expected. Specifically: Name should be SiteName and fi should be FiOrg. Also, the colon is missing after brokerid.

      Delete
  11. It is not a cut/paste since it is on a different computer. It is all entered properly in the form on the sites.dat file

    ReplyDelete
  12. Hi Robert! Got a question. For accounts that use the same bank, how hard would it be to set it up to get all files on the same login session? I have just changed the password on BoA where I have multiple accounts. I forgot to change the password on PocketSense, so it issued an incorrect login multiple times... boom. BoA locked me out. My fault entirely, but it would be less likely to happen if only one login was used for all accounts in the bank.

    ReplyDelete
    Replies
    1. I'm pretty sure the OFX standard supports it, but ... the scripts are written w/ an explicit "one connection per statement" logic. I like the idea, but I'd need to think it through. I'm certain that it can't be done "as is", and that some core logic would need to be revamped. The nice thing about "one connection per request" is that each account is processed independently, and therefore no assumptions re "grouping by sign-on credentials".

      Delete
  13. Has anyone been using Pocketsense with what used to be Hewitt Associates and is now Alight Solutions? I have been getting an error message "EOF occurred in violation of protocol" for about a month now, and I assume that they have changed things at their end. However, they have been remarkably and breathtakingly unresponsive in telling me what changed, so I wondered if anybody here had been using them and knew what to change to make it work again?

    Harold

    ReplyDelete
  14. In Windows Internet Options, in the Advanced tab, make sure that Use TLS 1.0, Use TLS 1.1, and Use TLS 1.2 are all enabled.

    CL

    ReplyDelete
    Replies
    1. I already had it set that way, but I tried turning it off and on... didn't fix it. Which means the problem still persists...

      Delete
    2. I upgraded to Active Python 2.7.13, and that solved the problem.

      Delete
  15. My BofA accounts stopped working this week (my last successful run was last week) with the error "An incorrect username/password combination has been entered. Please try again." Neither the username nor the password has changed and I was able to successfully login via their website as usual. I have a credit card and 3 checking accounts I use with PocketSense. The downloads have been working perfectly for a couple of years. Any ideas before I try to call BofA and see if my OFX access was disabled?

    ReplyDelete
  16. Same issue with BoA here. Last successful download was Sept 23.

    ReplyDelete
  17. Just a FYI for folks having issues... be sure you've updated to the latest Python 2.7.x version.

    ReplyDelete
    Replies
    1. Do you mean the updated scripts (from 8/27) or upgrading Python itself? I did download the 8/27 scripts yesterday and am still getting the same error.

      Delete
    2. TO ALL BOA CUSTOMERS:

      Change the ODXVER in Sites.dat to 103 and give it a go.

      The Bank is clueless and helpless as usual.

      Ray

      Delete
    3. I added this line to the BofA site profile in sites.dat but the error remained the same:

      ofxVer : 103

      Delete
    4. I was referring to Python itself, since there were one or two versions that caused SSL connection issues. Probably not related in this case, but a good idea.

      Delete
    5. I upgraded to Active Python 2.7.13, and that solved my problem with Hewitt Associates / Alight Soutions that began on 8/17/17 (presumably due to some kind of library upgrade on their side). Thanks!

      Delete
  18. Thank you Anonymous.

    ofxVer: 103 worked for me in solving BoA issues.

    It worked on my BoA personal accounts, BoA business accounts (different bankid) and BoA credit card. I'm using ActivePython 2.7.1.4

    Robert - are the order and spacing of the sites.dat entries critical? Don't know if that might have affected B-Man and why 103 didn't work for him.

    Here is my working entry:

    SiteName : BOA BANKING
    AcctType : BASTMT
    fiorg : HAN
    fid : 5959
    url : https://eftx.bankofamerica.com/eftxweb/access.ofx
    bankid : 052001633
    brokerid :
    ofxVer : 103
    appid :
    appver :
    mininterval: 30

    ReplyDelete
    Replies
    1. Field order doesn't matter, but probably a good idea to cross-check all parameters vs your working set.

      Delete
    2. I updated to ActivePython 2.7.13.27 and still same error. Here's my Sites.dat entry for BofA:


      SiteName : BOFACHECK
      AcctType : BASTMT #bank
      fiorg : HAN
      fid : 5959
      url : https://eftx.bankofamerica.com/eftxweb/access.ofx
      bankid : 053000196
      brokerid :
      ofxVer : 103
      appid : QWIN
      appver : 2300
      mininterval:
      timeOffset :

      Delete
    3. moneydance's config file is located at http://moneydance.com/synch/moneydance/fi2004.dict

      This is their entry for BofA:
      {
      "id" = "md:1039"
      "access_type" = "OFX"
      "fi_name" = "Bank of America"
      "bootstrap_url" = "https://eftx.bankofamerica.com/eftxweb/access.ofx"
      "fi_id" = "5959"
      "uses_fi_tag" = "y"
      "fi_org" = "HAN"
      "app_id" = "QWIN"
      "app_ver" = "2400"
      }

      It looks like you need to set appver to 2400. The bankid looks like it is the ABA routing number for your account, double check that yours is correct. I also do not use BofA.

      Delete
    4. What I had to do to get BofA downloading again:
      1. ofxVer : 103
      2. run Setup.py, test accounts, test BofA, it fails.
      3. Use my browser to login to bankofamerica.com, it will prompt you with a below message.
      4. Click OK, it will start the 10 minutes countdown.
      5. Go back to the command prompt where I have my Setup.py still running, test my BofA account again (within the 10 minutes). It works.

      --------------------------
      Thanks for signing in.


      We want to make sure you stay protected when you link your Bank of America account to financial management software.

      To stay secure, you’ll need to sign in to Online Banking when you:

      ◾Use Quicken or QuickBooks on an unrecognized device
      ◾Add a new Bank of America account to record transactions
      ◾Receive a software update for Quicken or QuickBooks

      Now that you’ve signed in, you can go back to Quicken or QuickBooks and continue to download your transactions.




      Please note: You’ll need to sign in to Quicken or QuickBooks within 10 minutes of selecting OK or you’ll have to repeat this sign-in process at another time.

      Delete
    5. Whew! I got the prompt when I signed in through BofA OLB and now the downloads are working. Thanks all!

      Delete
  19. Looking at the GnuCash site OFX_Direct_Connect_Bank_Settings page, you might try removing your appid or using "appid: QBW".

    I don't use BofA. Just an idea.

    CL

    ReplyDelete
    Replies
    1. Thanks for the continued suggestions all. Nothing's helping, though. Same error when I removed the appid and when I changed it to QBW.

      Delete
  20. Thanks to all for the help with the recent issue on Bank of America! Keep up the good work.

    ReplyDelete
  21. I got my BOA checking/savings accounts working by adding in the ofxVer : 103 line and then logging into the website and approving the access. However, I still can't get my BOA credit card to work. It just gives the old username/password error. Here is my entry.


    SiteName : BOACC
    AcctType : CCSTMT #credit card
    fiorg : HAN
    fid : 5959
    url : https://eftx.bankofamerica.com/eftxweb/access.ofx
    bankid :
    brokerid :
    ofxVer : 103
    appid :
    appver :
    mininterval:
    timeOffset :

    ReplyDelete
    Replies
    1. I've never been able to get BofA checking and savings accounts to download through PocketSense even though I've had no trouble with their credit cards. Can you tell me what OFX settings you're using that work for you?

      Delete
    2. Ken, I believe I added his BOA settings in my blog page.

      https://microsoftmoneyoffline.wordpress.com/2010/10/06/cal-learners-review-fidelity-401k-citi-card-and-vanguard-account-info/

      Delete
    3. Thank you! Your settings worked after I changed the BankID to the ABA number for my accounts. You may want to make a note that one needs to check the ABA number, which changes for different States. Otherwise the settings worked perfectly.

      Delete
  22. The CC is working now. I had not added the ofxVer:103 line until after I had done it on the checking/savings and tested those. When I ran it again they all errored out with the 'or log on' error so I logged on to the website, gave permission, and now everything works.

    ReplyDelete
  23. Another BoA update. ofxVer initially worked for me, and then today I started getting invalid username/password errors. Login on via the web site gave me the Quicken permissions prompt, and then everything went back to normal.

    Bottom line - there is something BoA is doing which may cause you to loose Quicken access. So first order of defense, if PocketSense fails, immediately log unto web interface to make sure you give permissions.

    I'm not sure if they are now sensitive to IP changes or some other variable. This might make sense, since they are effectively trying to set up a two-step initial authentication.

    ReplyDelete
  24. I started getting this error for my Citi Card account:
    ** An ERROR occurred sending POST request to www.accountonline.com
    Exception type :
    Exception val : [Errno 1] _ssl.c:507: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version
    An online error occurred while testing the new account.

    Anyone else?

    ReplyDelete
    Replies
    1. Citicard works for me. It worked as scheduled overnight, and I just tried again as a test using setup.py.

      My site descriptor is simple:
      ==========

      SiteName : CitiCard
      AcctType : CCSTMT #credit card
      fiorg : Citigroup
      fid : 24909
      url : https://www.accountonline.com/cards/svc/CitiOfxManager.do
      bankid :
      brokerid :
      appid :
      appver :
      mininterval:
      timeOffset :

      ================

      Delete
    2. That has been my setting too, but since Sunday it hasn't worked. All of my other accounts are working though.

      Delete
    3. I get the same error on my Citicard and Searscard, also since Sunday...

      Delete
    4. I have a Universal AT&T card, which is also Citibank, and uses the same server. Just tested, and works fine. Site settings are the same as above.

      The error listed above is consistent w/ what's been seen when running an older version of Python, but the server updates to a newer version, which triggers an SSL exception. So, the first thing is to verify that you're running the latest Python release [2.7.13].

      Delete
  25. Thank you so much for the reply Robert! I was at 2.7.6 and I’ve upgraded to 2.7.14, which is apparently the latest as of Sept 16th. It resolved the issue!

    Dan (ameridan)

    ReplyDelete
  26. Discover Bank has found yet another new way to screw up their FITIDs. The latest new format is:
    YYYYMMDDSDFCCCCCCXXXXXXXX

    YYYY year posted
    MM month posted
    DD day posted
    SDF literal
    CCCCCC six digit non-unique code. two checks with different amounts posted on same day have same code
    XXXXXXXX eight digit sequence that changes daily (transaction has different sequence on next day)

    Below is a context diff of the change required to get Discover Bank working again. This change is
    in _scrubDiscover_r1() in scrubber.py. NOTE: blogspot strips the required indentation

    #strip the serial value for credit card transactions
    if accType=='CCSTMT':
    bx = len(fitid) - 5
    fitid_b = fitid[:bx]
    + #strip the serial value for bank transactions
    + else:
    + bx = len(fitid) - 8
    + fitid_b = fitid[:bx]

    #find a unique serial#, from 0 to 9999
    seq = 0 #default
    while seq < 9999:
    fitid = fitid_b + str(seq)

    ReplyDelete
    Replies
    1. Thank you Andrew. I made the changes, and then realized I only use Discover Card, not Discover Bank, so I am not seeing the problem. Nevertheless, the scrubber still works.

      Delete
  27. Following are experimental changes I have made to populate the check number field for check withdrawals.

    In _scrubDiscover() in scrubber.py. NOTE: blogspot strips the required indentation and "{" and "}" need to be replaced by "<" and ">"

    _sD_knownvals = [] #reset our global set of known vals (just in case)

    + # insert a line break before each transaction for readability
    + # this should also ensure that the regexes below don't match across transactions
    + p = re.compile(r'({STMTTRN})',re.IGNORECASE)
    + ofx_final = p.sub(lambda r: _scrubDiscover_r0(r, accType), ofx)
    + ofx = ofx_final
    +
    #regex p captures everything from {FITID} up to the next , but excludes the next "{".
    #p produces 2 results: r.group(1) = field, r.group(2)=value
    p = re.compile(r'({FITID})([^<\s]+)',re.IGNORECASE)

    #call substitution (inline lamda, takes regex result = r as tuple)
    ofx_final = p.sub(lambda r: _scrubDiscover_r1(r, accType), ofx)

    + #regex p captures everything from {TRNTYPE}DEBIT up to the next "<" after the {NAME}Check tag and field.
    + # Discover Bank codes checks as
    + # {STMTTRN}{TRNTYPE}DEBIT{...}{NAME}Check############{/STMTTRN}
    + #p produces 4 results:
    + # r.group(1) = {TRNTYPE}DEBIT,
    + # r.group(2)=stuff up to next "{NAME}Check "
    + # the \s keeps it from matching across transactions due to the added \n since regex will try to make the longest match
    + # r.group(3) = "{NAME}Check ",
    + # r.group(4) is the check number
    + # The result should produce a entry that will import the check number in Money
    + # {STMTTRN}{TRNTYPE}CHECK{...}{CHECKNUM}############{NAME}Check{/STMTTRN}
    + if accType=='BASTMT':
    + ofx = ofx_final
    + p = re.compile(r'({TRNTYPE}DEBIT)([^\s]+)({NAME}Check[ ]+)([0-9]+)',re.IGNORECASE)
    + #call substitution (inline lamda, takes regex result = r as tuple)
    + ofx_final = p.sub(lambda r: _scrubDiscover_r2(r, accType), ofx)
    +
    return ofx_final

    Two new added functions

    + def _scrubDiscover_r0(r, accType):
    + fieldtag = r.group(1)
    + return '\n' + fieldtag
    +
    + def _scrubDiscover_r2(r, accType):
    + #regex subsitution function for _scrubDiscover()
    +
    + trntype = r.group(1)
    + rest = r.group(2)
    + name = r.group(3).strip(' ')
    + checknum = r.group(4)
    + return '{TRNTYPE}CHECK' + rest + '{CHECKNUM}' + checknum + name
    +

    ReplyDelete
  28. Thanks Andrew. If you have a chance, can you send your new scrubber.py module to pocketsense1 /at/ gmail /dot/ com? I won't be able to look at it for a couple days, but that will save reformatting. I don't use a Discover Checking account, so I can't test myself.

    ReplyDelete
  29. It looks like Yahoo is futzing with there CSV server. It's throwing an error 'An error occurred connecting to the Yahoo CSV service'.
    -Kevin N.

    ReplyDelete
    Replies
    1. I noticed that recently, but no issues w/ the Google service.

      Delete
  30. How do I get the Google service to work? I have "EnableGoogleFinance: Yes" but it does not solve the problem. Thank you.

    ReplyDelete
  31. I found my prices were up to date. I had thought those were coming from Yahoo. Maybe not.

    In my sites.dat, I have these:
    ----------
    #YahooURL: http://ca.finance.yahoo.com
    QuoteCurrency: USD
    EnableYahooScrape: Yes
    YahooTimeZone: -5:EST
    ShowQuoteHTM: No
    AskQuoteHTM: Yes
    ---------------------------

    My *.py files are dated august 2017. That indicates that I have the latest released Pocketsense. I don't know why my prices that are not from OFX files are working, but of course I am glad they are.

    CL

    ReplyDelete
  32. Like Kelvin I get "An error occurred connecting to the Yahoo CSV service". My sites.dat is similar to CL's except #QuoteCurrency: USD.
    I also tried quotes.py from Harry Sit (The Finance Buff) but no luck
    GG

    ReplyDelete
  33. I am seeing the same thing. Last good update of stock prices was yesterday, 11-1
    DD

    ReplyDelete
  34. I plugged the url into my browser and got the following message: It has come to our attention that this service is being used in violation of the Yahoo Terms of Service. As such, the service is being discontinued. For all future markets and equities data research, please refer to finance.yahoo.com.

    ReplyDelete
    Replies
    1. UPDATE - just turn off the Yahoo site in the sites.dat file. It tries Yahoo first if both Google & Yahoo are enabled. By turning off Yahoo it goes to Google and works fine.

      Delete
    2. Unfortunately, the Google finance page often has very wrong prices or the lookup for a particular ticker (especially leveraged ETFs and not so well known ETFs) goes to the wrong equity requiring corrections. Perhaps the Yahoo code can be replaced with Yahoo web page scraping rather than the web API. There are no restrictions on scraping their finance page itself.

      Delete
    3. I see that the sites.dat file has
      EnableYahooScrape: Yes # Try a screen scrape if the primary Yahoo csv

      Shouldn't this work if CSV fails?

      Delete
  35. Regarding quote lookup on Yahoo and Google:

    1. There is currently a Yahoo screen scrape routine in the scripts, should the csv interface fail. However, the scrape routine doesn't work w/ the current Yahoo site. Look in the sites.template for the latest available settings, but the following two are important here:

    EnableYahooFinance: No # Enable quote lookup on Yahoo Finance
    EnableGoogleFinance: Yes # Enable quote lookup on Google Finance

    2. I've personally had no issues w/ Google Finance quotes, and their database has proven much more extensive. However, the symbols used by Google don't always match what is used by Yahoo. You must lookup the actual ticker you want, and then get the exact symbol from the header. For example, the S&P500 ticker on Google is "INDEXSP:.INX", or more simply ".INX". To be precise, you can use the fully qualified Google ticker, but for common US tickers you don't need the prefix.
    3. If you need to map a Google symbol to an existing symbol in Money, you can use the s: option when defining the ticker. Using the example in #2, you would add "INDEXSP:.INX s:^GSPC", so that the quote is grabbed for the fully qualified ticker, but "^GSPC" is passed to Money as the symbol.

    I'll try to look again at the Yahoo scrape option, and see where it stands. I'm about to put out a patch for the recent Discover Bank numbering issue, and may be able to include this as well. The thing to remember about scraping is that it's UNRELIABLE unless the site uses consistent/reliable encoding. Yahoo previously did not, but on first glance, they may have changed their ways (?).

    ReplyDelete
    Replies
    1. Ok thx. Many symbols need to be fully qualified with the exchange prefix for using Google Finance. For example, NYSEARCA:FSTA. Just using FSTA for the ETF will download for Fuller, Smith and Turner PLC quote. Similarly unqualified SHYG will download for SHYG on the London Stock Exchange, not the ETF on NYSE. Mutual funds seem to work OK for the most part as will most 5 letter symbols because there is no ambiguity across exchanges.

      Understand the issue with scraping. Hope there is a solution to avoid having to sanitize every symbol as above but if that is what is needed ....

      Delete
  36. Robert, Thanks for this great answer. I had never tried the scraping option. Do the currencies work with GoogleFinance? I set EnableYahooFinance to No, and currencies failed.

    ReplyDelete
    Replies
    1. Take a look at finance.google.com . On the right, you should see some example currency quotes. Click the link, and you'll see the ticker in the header. Example: EUR/USD ticker is EURUSD. The same pattern is used throughout their currency tables (i.e. USD/JPY == USDJPY). Should be mostly the same as Yahoo, except Yahoo had a "=X" suffix, iirc.

      Delete
  37. This comment has been removed by the author.

    ReplyDelete
  38. Thanks Robert, for the currency tip. I didn't realize =X was part of the Yahoo quote and not a special code of some sort.

    I'd also forgotten that currencies get downloaded as a "quote" and don't update Money's currency info. Oh well.

    In any case I can't believe that seven years later we are still able to use Money thanks to your scripts. On the other hand, not everything else is as lasting. You may want to remove YHOO from the sites.dat template :)

    -- Carlos

    ReplyDelete
  39. Hi Robert,

    I noticed the following at Google.finance regarding personal portfolios...

    "Google Finance is under renovation. As a part of this process, the Portfolios feature won't be available after mid-November 2017. To keep a copy, download your portfolio."

    Do you think that this may have any effect on the scripts' ability to retrieve stock quotes from Google?

    -Kevin N.

    ReplyDelete
    Replies
    1. Should be fine, but "we shall see".

      Delete
  40. I have been using Pocket Sense for years. It stopped working last week. I was only building a QFX file to import into my 'Qoutes' account. I am willing to pay for some help (via paypal). I don't understand you to use this forum or start a new thread.

    ReplyDelete
    Replies
    1. See post above on 03-Nov-2017 describing how to enable Google quotes and *disable* Yahoo. The Google Finance option works well for situations I've tried. It was implemented specifically because it provided quotes for a broader range of global exchanges, and as a backup for the Yahoo service.

      If you want to wait a bit longer, I'm looking at updating the Yahoo quote interface to work w/ their new finance site. I haven't had time to focus on it, so can't say for sure how it'll go or when, but it seems promising.

      Delete
    2. Still lost. I have been using an OLD version of Pocket Sense. It failed since Yahoo changed so I am trying to start over. I am trying to

      follow your instructions "OPTIONAL: Create an Account in Money to Receive Stock Quotes". In Money my dummy account is 'Quotes'.I turned off EnableYahooFinance (and YahooScrape) and I can collect prices BUT only if I do it from Setup.py but it will not build a good QFX file. If I try to start GetData.py I get "No accounts have been configured. RUn SETUP.py to add accounts. Do I need to create a 'Quotes' account to match my Money file? I have tried to follow the instructions but I just don't get it.

      If you think you can help me then let's use TeamViewer to get the job done. I will gladly buy you a stake dinner for two.

      Delete
    3. #1: Enable Quotes in Setup (option 6).

      Delete
  41. Some of the mobile apps that also got perturbed by Yahoo change have switched to https://iextrading.com/developer/docs/#iex-api-1-0
    for quotes. Perhaps, that is more robust/stable than scraping Yahoo web site.

    ReplyDelete
    Replies
    1. Pretty sure their api only works w/ stocks traded on the IEX exchange, which is a small universe. Yahoo and Google are hooked into all the exchanges, mutual funds, commodities, etc.

      Delete
  42. There is a problem with quotes from Google for mutual funds. The date of the quote is wrong and so it doesn't update Money prices properly. For example, all mutual funds right now are reporting as Nov 5. You can verify this by going to the Google Finance page as well and noticing the date below the quote. How is this working for others? Or am I doing something wrong?

    ReplyDelete
    Replies
    1. Hi,
      I'm seeing the same thing. At first I thought that these were 2-day-old quotes but as you've discovered, the quotes appear to be current but the date is off.
      -Kevin N.

      Delete
    2. I see the same, but it only seems to apply to class=MUTF quotes (US mutual funds)? The price is the latest, but the quote time is a day behind. Money should still reflect the "latest" value, though, since it will use the "last quote" for pricing, regardless of the date.

      It would be easy to correct in the scripts, *but*, it's a bug in the Google code that's likely to get fixed. If so, the script would then be wrong :(

      Delete
    3. Hi Robert,

      You're correct, the date discrepancy appears to only apply to Mutual Funds.

      I've only recently started using Google for quotes. Like you say, the values should still reflect current values regardless of the date.

      By a 'fix' do you mean something similar to the current m:value or s:value options? If so, that switch could be shut off if and / or when Google squashes their bug.

      Then again, if the values are correct then there isn't much need for a fix. Carry on. :)

      -Kevin N.

      Delete
    4. Unfortunately, Money is NOT updating to the "latest value" for me requiring a manual update for every MUTF. I wonder if the fact that there is a manual update already prevents the update a day or two later with that same date. For example, I manually corrected the fund prices for Nov 11 yesterday in Money. Now, if I update at the end of the day today and Google, say for example, reports the date as Nov 11 but with Nov 12 prices, will Money update the value for Nov 11 that already exists with the Nov 12 price? I don't think this is happening. There may be something else wrong (say what time the script is run) but I am definitely not seeing (from looking at the reported price as well as the price history in manual price update dialog) Money update the prices to current value but with a previous date. So, it requires a manual update every time.

      Delete
    5. Money will not overwrite an existing quote entry for a given date during import. If a person runs the script ten times in a day, for the same date, it only imports the first one. If you enter a quote price manually for a given day, it will not be updated by an import.

      Delete
  43. So if I'm trying to get good settings in Sites.dat, what do I need to do between tests?

    In other words - try settings a, b, and c. Failure, going to change c to d.

    Assuming in that case that the failed new account needs to be deleted and recreated, but do I also need to exit and relaunch setup.py?

    ReplyDelete
    Replies
    1. Account info configured within Setup will update as you go. Data changed in sites.dat is loaded when setup first starts.

      Delete
    2. Thanks Robert. Do you mean that if I change a fiorg in the sites.dat, exit and relaunch setup, and test an existing account that used that entry it would update and use the new fiorg?

      Delete
  44. >So if I'm trying to get good settings in Sites.dat, what do I need to do between tests?

    >In other words - try settings a, b, and c. Failure, going to change c to d.

    >Assuming in that case that the failed new account needs to be deleted and recreated, but do I also need to exit and relaunch setup.py?

    To simplify things, I would use the test feature of setup.py. Select to Test an account. Then choose the number for "Stock/Fund Prices" as the account.
    Test Stock/Fund Pricing Updates (Y/N)? y
    Open in the default browser (y/n)?y
    Send the results to Money (y/n)? n
    With that "n" for sending the results to Money, your money file will not be affected. Test as many times as you want. Your Money file will not be affected if you choose to not send the results to money. Money generates a new quotes*.ofx and a quotes.htm each time you do that. At least I think that is how it works.

    CL

    ReplyDelete
    Replies
    1. Other way around - when is sites.dat read? Do I have to exit and relaunch for my changes to be reflected?

      Or am I looking at it wrong and sites.dat is always reread and I don't even have to delete the account, just make my changes to the file in notepad and just try the test again to see the changes if my acct number/user/pass didn't change?

      Delete
    2. As long as you save the file after editing, yes, it is reread.

      Delete
  45. For quotes, it seems that Yahoo's json api is still working
    https://www.bogleheads.org/forum/viewtopic.php?f=10&t=218779&newpost=3609317&view=unread#p3609317

    ReplyDelete
  46. >Other way around - when is sites.dat read? Do I have to exit and relaunch for my changes to be reflected?
    To test this, I added an additional stock to the list in sites.dat, without closing setup.py.
    I again select to test the quotes. The values got updated, and the new stock was added to the list of fetched quotes.

    CL

    >Or am I looking at it wrong and sites.dat is always reread and I don't even have to delete the account, just make my changes to the file in notepad and just try the test again to see the changes if my acct number/user/pass didn't change?

    Fetching quotes does not involve what I would call a real account. For broker or credit card accounts, you are fetching OFX files. Yes OFX files can include prices for securities held. My experience is that these are only update by the broker with closing prices and those updates are posted during the night.

    For testing out account access, rather than quotes, I would look at the OFX files in XFR without sending them on to Money. A bad password will result in a small OFX file. Other errors would result in small files I would think also. Expect that file to have the word ERROR in it.

    ReplyDelete
  47. Hi Robert,
    In your quest of fixing the Yahoo interface, I would like to bring to your attention that Yahoo has a json api which seems to be working:
    url = "http://json-csv.com/?u=http://query1.finance.yahoo.com/v7/finance/quote?symbols=%s" % self.ticker

    Someone at the bogleheads forum posted his/her Python code to retrieve quotes through this interface:
    https://www.bogleheads.org/forum/viewtopic.php?f=10&t=218779&newpost=3609317&view=unread#p3609317

    You might want to take a look.

    I posted this information yesterday: AnonymousNovember 8, 2017 at 2:23 PM but it was mixed up with another thread.

    ReplyDelete
    Replies
    1. Thanks, and yes, I have the json interface on the short list, and have a parser ~ ready.

      I'm skeptical of anything that has a version# in the path, however ;) So... I was looking at adding a parser for the regular site as a backup. They now encode quotes into the web page (json also), but it's scattered, and filled w/ similar info for other quotes being displayed.

      I've been short on time, so may just put out the json data api, and add the site backup later. They may use the json interface for their own feeds, so if they keep the basic format the same, the sites.dat YahooURL option would allow folks to update the script setting if/when the Yahoo path changes.

      Delete
  48. I added a new post describing recent changes and issues, so I'm going to close out comments on this one.

    ReplyDelete